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ABSlfiACT 

This report of the developaent and validation of a 
criiterioc-ref erenced exaaination for physician's assistants contains 
an overview of phase 1 project activities (1972-7U) and aiore detailed 
description aud data on phase 2 activities (1975-76)* The overview 
describes the rationale for the developaent of the exaaination* (It 
was the first certifying exaaination for health care professionals 
that was based on task analyses of health care functions — data 
gathering, analysis and interpretation, and aedicail and health care 
strategies--as opposed to curricular content*) Adainistration of the 
first National Certifying Exaaination for Priaary Health Care 
Physician's Assistants in 1973, the validation studies of that exaa, 
and the adainistration and statistical analyses of the 1974 exaa are 
suaaarized in the overview* Phase 2 activities are described in 
detail in three sections: (1) A study of audiovisual siaulation 
technigues for assessing interpersonal sJcills, (2) a validation study 
of the 1974 certifying exaaination involving aedical record audits, 
and (3) an analysis of the 1975 certifying exaaination. Appendixes A, 
B, and C contain the audiovisual siaulation aaterials which include 
the saaple paper-pencil siaulation, the interaction observation fpra, 
and the patient protocol fora* Appendixes D and E contain the audit 
validation study materials, which include the saaple audit criteria 
set and the instruction manual and saaple abstracting fora for record 
reviewers. References are provided* (TA) 
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OVERVIEW OF PHASE I ACTIVITIES 



BACKGROUND ON EXAMINATION DEVELOPMENT 

When work on tins examination program was initiated in July, 1972, two primary goals were 
identified: (a) to specify the roles and responsibilities of tiie primary care physician's 
assistant, and (b) to identify evaluation techniques that could be used to assess competence 
in performing tlie functions of a physician's assistant. The development of examinations for 
health care professionals had long been based upon curricular content: the subject matter 
presented and the way in which its presentation was structured in the curriculum. However, 
because of the goals of tiiis examination program, an approach was devised which had not 
previously been used for the development of national examinations for health care professionals. 
Instead of relying upon descriptions of the subject matter content within the various educational 
programs, a task inventory consisting of several hundred healtii care functions was designed. 
These functions were compiled from several task analysis studies of physician and non-physician 
activities. Once compiled, each health care function was classified under one of ten categories 
reflecting the various components of the clinical problem-solving and management process. These 
categories are listed below: 

I. Data Gatliering 

A. History Taking and Patient Records 

B. Pliysical Examination 

C- Laboratory Tests and Investigative Procedures 
D. Patient Monitoring 

n. Analysis and Interpretation 

A. Consultation and Referral 

B. Diagnostic Acumen 

ni. Medical and Health Care Strategies 

A. Emergency Procedures 

B. Surgical and Technical Procedures 
C- Management 

D. Patient Counseling 

The resulting task inventory was distributed to' each member of the National Board Advisory 
Committee on Physician's Assistants whose membership included physicians involved in the 
development of this new category of health care personnel, physician's assistants, physicians 
who were training and utilizing physician's assistants, and nurses concerned with the nurse 
practitioner concept. These individuals were asked to consider each of the 900 health care 
functions included in the inventory, and to decide whether the health care function was one that 
a Type A primary care physician's assistant should definitely, probably, probably not, or 
definitely not be skilled in performing. Since the examination to be developed would be 
administered on a national basis and would be evaluating individuals trained in different types 
of educational programs, it was felt that the examination should be designed to measure those 
health care functions that a representative group of experts felt were ones the physician's 
assistant should definitely be skilled in performing. In order to identify the health care 
functions about which there was such a consensus, a frequency distribution of the twenty 
judgments made concerning each health care function was tabulated for all functions listed 
on the task inventory. Each of the four rating categories was given a numerical value on a 
scale of one to four a value of four being assigned to the 'definitely" category, and a value of 
one to the "definitely not" category. In turn, each judgment made by a rater was given in 
numerical value on tlic basis of tlie column into which a check mark had been placed. 
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The arithmetic mean of these judgments wns then computed for each health care function. Those 
functions receiving a mean value of 3. 5 to 4.0 were considered as ones that, in the view of expert j 
opinion, a Type A primary care physician's assistant should definitely be skilled in performing. 
Functions receiving a mean value of 3.40 to 3.49 were reviewed by this expert group at a meeting 
during which the results of the task inventory study were presented. 

A review of the health care functions identified from tliis task inventory study suggested that not all 
functions were equivalent in terms of their importance to the proficiency of the primary care 
physician's assistant. Moreover, the number of functions was so large that it was evident that no 
examination program could attempt to sample adequ.'^tely the knowledge and skills related to all of 
tJiem. For tJiese reasons, a priority study was conducted using tne same experts who had partici- 
pated in the Lisk inventory study, plus four additional experts, all of whom were pediatricians. 

The purpose of the priority study was to establish the relative importance of the several hundred 

health care functions, so that those receiving the highest rating )n tins regard could provide 

tJie primary focus for the development of tJie Certifying Examination for Primary Care Physician's 

Assistants. 

Two dimensions were selected as the basis for determining the priority of each health care function* 

(1) the frequency witii which the function might be carried out in a primary care practice, and 

(2) its 'triticalness" to optimum healtli care delivery. In order to determine the priorities of 

the several hundred health care functions identified from the first phase of this process, a second 
task inventory was prepared using the categories previously described. The two dimensions which 
were tiie basis for arriving at these priority ratings (i. e. , frequency and criticahiess) were 
represented by t^vo scales each consisting of four intervals. The anchor points of each scale were 
labeled "high" and "low''. Each member of the group of expert judgr>s was asked to consider the 
functions included on this second inventory, and to indicate how frequently he felt the task would be 
performed in a primary care practice and how critical it was to optimum health care delivery. 

A frequency distribution of the judgments made regarding tlie frequency and criticalness of each 
health care function was tabulated. A mean value for each health care function on each dimension 
was calculated by assigning a numerical value of one to four to the intervals on each scale as was 
done in tlie first task inventory study. 

The priority value assigned to each health care function was determined by using the" following 
formula P = f f 2(c), where f was the mean frequency value and c the mean criticalness value. 
The criticalness value was weighted more heavily than the frequency value because it was 
recognized tiiat while some functions are performed infrequently (e.g., closed chest cardiac 
massage), they often involve life-and-death implications when they are required as part of health, 
care delivery. The use of the preceding formula resulted in a scale whose range was 3.00 to 12 00. 
Those functions receiving a priority of 8.00 to 12.00 were identified as high priority health care 
functions and were further analyzed in tlie following manner. 

Since the validity of an examination as an assessment of proficiency depends upon its capacity Lo 
evaluate accurately the knowledge and skills required to carry out specific healUi care functions, 
test committees were appointed to analyze related health care functions and to identify the knowledge 
and skill components related to each. These knowledge and skill components were stated behavior- 
ally to facilitate the selection of appropriate evaluation methodology and to serve as the performance 
criteria to be used in assessing the proficiency of physician's assistants Table 1 presents some 
of the criteria specilied for the following health care functions: history-taking; physical examination; 
wound care; suturing lacerations; taking an electrocardiogram; and applying a plaster case. (A more 
detailed description of tliis phase of examination development is provided in the Project Report 
1973-74.) 



Table 1 

SAMPLE CRITERIA FOR SELECTED HEALTH CARE FUNCTIONS 



Function 




Criteria 


History Taking 


(a) 
(b) 


Elicit pertinent positives and negatives related 
to chief complaint; 

Elicit pertinent data from review of systems. 


Physical Examination 


(a) 
- (b) 


Recognize and name common skin lesions; 
Recognize and describe abnormalities seen 
on funduscopic examination. 


Wound Care 


(a) 
(b) 


Recognize normal signs associated with 
wound healing; 

Identify lacerations requiring primary 
closure by suture. 


Suturing Lacerations 


(a) 
(b) 


Select appropriate suture materials; 

Effect a closure in which all dead space has 

been obliterated. 


Electrocardiogram 


(a) 
(b) 


Identify the correct placement of all "V" leads; 
Select appropriate courses of action to 
eliminate or minimize technical defects in an 
EKG tracing. 


Cast Application 


(a) 
(b) 


Identify the correct position for applying a 
forearm cast; 

Recognize errors made in case application- 



ADMINISTRATION OF THE FIRST CERTIFYING EXAMINATION - 19 73 

The first National Certifying Examination for Primary Care Physician^s Assistants was 
administered on December 12, 1973, to 880 candidates in 38 test centers across the country. 
Eligibility for the first examination was limited to graduates of primary care physician's 
assistant training programs tliat were approved by the AMA Council on Medical Education, 
funded by the Bureau of Health Resources Development, or, in the cast; of nurse practitioners, 
programs of at least four months' duration within a nationally accredited school of medicine or 
nursing that trained pediatric or family nurse practitioners. 

Of the total number of physician's assistants eligible, 75 per cent registered for this examination, 
while almost 100 per cent of the eligible Medex did so. Of the nurse practitioners who were 
eligible for the examination, approximately 10 per cent registered. 

In relation to the 880 candidates who took this examination, 62 per cent had received their 
training in physician's assistant programs, 29 per cent in Medex training programs, and 
9 per cent in nurse practitioner programs. Table 2 summarizes the composition of the examinee 
group by type of training program. 
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Table 2 

COMPOSITION OF EXAMINEE GROUP - 1973 



Type of Training 


No. 


% 


Physician's assistant 


538 


62 


Medex 


265 


29 


Nurse practitioner 


77 


__9 


Total 


880 


100 



Biographic data collected on each examinee indicated that a significant majority had already 
completed their formal training prior to the examination, and had already acquired clinical 
experience as a physician's assistant. Table 3 summarizes the clinical experience of this 
examinee group. As illustrated in the table, 89 per cent of the examinees had already 
completed aii educational program, and 81 per cent had already acquired postgraduate clinical 
experience prior to taking Uie Certifying Examination. Of those examinees who had already 
acquired experience as a physician's assistant, 88 per cent had had as much as two years of 
clinical ex]Derience, while 12 per cent had acquired more than two years of clinical experience 
as a physician's assistant. 

In addition, 91 per cent of Uie total examinee group had acquired experience in health care 
delivery prior to tlieir training as a physician's assistant. Of tins group with prior experience 
in health care delivery, 86 per cent had been involved in direct patient contact (for example, 
as a nurse, military corpsman, or physical therapist) , while 11 per cent had been involved in 
health care delivery in a technical capacity. These data indicate that the typical examinee for 
the 1973 Certifying Examination had already completed a formal educational program and had 
already acquired several months to several years of clinical experience as a physician's 
assistant. Moreover, prior to having been trained as a physician's assistant, the typical 
examinee had been involved in direct patient contact health care delivery for a period of from 
two to four years . 

Table 3 



EXPERIENCE OF EXAMINEE GROUP - 1973 



Biographic Data 


% - Total Group 




Completed an educational program 


89 




Clinical experience since training: 


81 




Up to 2 years clinical experience 




(88) 


More than 2 years clinical experience 




(12) 


Prior exTDerience in health care delivery: 


91 




Patient contact 




(86) 


Technical 




(11) 


Other 




( 3) 
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The 1973 examination program consisted of a one-day written examination divided into two 
sections. The first section contained multiple-choice and other objective format questions 
presented in printed and pictorial form. These materials were desigTied to assess Uie candidate's 
knowledge and skill in applying knowledge related to high priority health care functions that a 
primary care physician's assistant should be skilled in performing. Items on this section of the 
examination were classified under the following headings: 

1. The identification and classification of physical findings; 

2. Patient management; 

3. Patient counselijig and instruction;' 

4. Knowledge related to clinical procedures (e.g., wound care, fracture management, 
cardiopulmonary resuscitation, electrocardiograms) . 

The afternoon section of the examination involved a programmed testing technique in which the 
candidate was presented with a simulated clinical case and asked to make decisions regarding 
the appropriate diagnostic work-up and management of tlie patient as he would in an actual 
clinical sotting. These patient management proDlems are designed to assess the candidate's 
skill in gathering pertinent information about patients and in making appropriate disposition 
decisions. Clinical cases were presented in both adult and pediatric medicine, and included 
emergency as well as non-emergency problems. 

Statistical analysis of the examination indicated that it was reliable and moderately difficult* 
for the group of examinees who took it. The reliability of the total examination was .89, 
which placed it witJiin the range of reliabilities for other National Board examinations. The 
mean difficulty level for the multiple-choice question section of the examination equaled .64 
which was also witJiin tJie range encountered on otJier National Board examinations. The mean 
difficulty level of the patient management problems was calculated at . 79 which also corresponds 
to the difficulty levels found on other examinations using patient management problems. These 
examination statistics are summarized in Table 4. 



Table 4 

EXAMINATION STATISTICS - 1973 PHYSICIAN^S ASSISTANT CERTIFYING EXAMINATION 



Range/Other Range/Other 

Examination NBME Mean NBME 

Component Reliability Examinations Difficulty Examinations 

Total .89 (.88-. 91) 

Multiple -Cjhoice questions .64 (.60-. 65) 

Patient management problems .79 (. 75-. 85) 



Further analysis of examination performance in relation to biographic data indicated that 
examinees who had already completed a training program and had acquired clinical experience 
as a physician's assistant or nurse practitioner scored significantly higher on the examination 
than did examinees without such postgraduate clinical experience. This finding provides evidence 
of the construct validity of the examination since it appears to be measuring knowledge and skills 
that are relevant to practice and that increase with clinical experience- 
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Table 5 presents the intercorrelations of tlie various examination components. The correlation 
coefficient of . 51 between tJie multiple-choice (MCQ) and patient management (PMP) sections of 
the examinations indicate that, altliough these components were assessing some overlapping areas 
of knowledge and skills, each was assessing aspects of competence tliat the oUier examination 
component was not, Tliis correlation coefficient is similar to those encountered on other 
examinations for licensure and certification and adds further support to the tenet Oiat these l^'o 
different examination formats do assess different aspects of clinical competence. 

Table 5 

INTERCORRELATIONS OF 1973 EXAMINATION COMPONENTS 



, (PMP) (PMP) 

Data Management/ (PMP) 
MCQ Gathering Therapy Total 

MCQ ~ ,46 ,45 ,51 

(PMP) Data GaUiering — ,36 ,98 

(PMP) Management/Therapy — . 55 

Total -~ 



Items on the patient management problems (PMP) were divided into two categories: (1) data 
gatliering procedures (history, physical examination, and laboratory tosts); and (2) management/ 
therapy decisions. The correlation behveen these tv^'o components of ihe patient management 
problems equaled .36, and indicates that items in each category are also assessing different 
aspects of clinical competence. 



VALIDATION STUDIES OF THE 1973 CERTIFYING EXAMINATION 

In order to investigate the validity of this examination program, a number of studies were 
conducted and analyses performed. The first evidence of construct validity was provided by 
an analysis of the examination itself in relation to certain biographic data that had been collected 
on all examinees. The fact that examinees who had already completed a training program and had 
acquired clinical experience as a physician's assistant or nurse practitioner scored significantly 
higher on the examination than did examinees without such postgraduate clinical experience 
suggests that the examination was measuring knowledge and skills relevant to clinical practice 
that increase with clinical experience. Since examinees with experience in patient contact 
health care delivery prior to training as a physicir.n'R fiobistant did not score significantly 
higher thari individuals witliout such prior experience suggests that the examination was 
measuring aspects of competence that are specifically pertinent to Uie proficiency of a physician's 
assistant or nurse practitioner. 

In addition to this internal evidence of validity, two external studies were also conducted, each 
focusing on somewhat different aspects of the validity of this examination program. 

Having already found evidence to suggest tliat the examination was measuring knoH'lcdge and skills 
relevant to actual clinical practice, a study was conducted in order to determine whether or not 
the examination was measuring components of competence that could be attributed to the training 
process itself. If the Certifying Examination was measuring knowledge and skills that could be 
acquired only tlirough a training program for physician's assistants, then one would expect 
examinees who sat as candidates for the 1973 Certifying Examination to score significantly 
higher than individuals who were just beginning a training program an^^vho, presumably, 
had not yet acquired the same level of proficiency. 
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In order to investigate this aspect of tlie construct validity of the examination, tlie 19 73 
Certifying Examination was administered to groups of individuals who were just beginning 
training as physician's assistants, Medex, or nurse practitioners. A stratified random sample 
of 16 training programs was selected to be representative of the types of educational programs 
in which actual candidates for the 1973 examination had been trained. As each program began 
training of a new class, the 1973 Certifying Examination was admmistered to the group of 
trainees. The same procedures used in administration of the actual Certifying Examination 
were also used in administering the examination for validation purposes. 

The validation study sample consisted of 357 examinees: 83% physician's assistants, 15% Medex, 
and 3% nurse practitioners. The actual candidate group in 1973 consisted of 880 examinees: 
62% physician's assistants, 29% Medex, and 9% nurse practitioners. The validation study 
statistics, however, were generated by comparing the validation sample of 357 examinees witli 
candidates for tiie 1973 Certifying Examination who had come from the same educational programs 
that participated in tlie validation studies. This actual candidate group numbered 528, 

A statistical com.parison of the performance of tb^^se two groups was carried out by performing 
t-tests on the total examination score as well as on scores obtained on the three components of 
tlie written examination. With respect to overall examination performance, the mean score of 
examinees from the validation study sample was 374, while a comparable sample of examinees from 
the actual 1973 examination obtained a mean score of 497. This difference between means is 
highly significant (p < .001) and indicates that individuals who had completed or were nearing 
completion of their formal educational program scored significantly higher than individuals who 
were just begmning training as a physician's assistant. Similar differences in performance were 
also observed on each of the three components of the written examination: multiple-choice 
questions, PMP data gaUiering, and PMP management/therapy , Thus, tlie 1973 Certifybig Exam- 
ination was able to distinguish between individuals with and without formal training as a physician's 
assistant. These statistics are summarized in Table 6 shown below. 

Table 6 

EXAMINATION SCORES OF INDIVIDUALS BEGINNING 
FORMAL TRAINING (UNTRAINED) AND CANDIDATES FOR THE 
1973 CERTIFYING EXAMINATION (TRAINED) 



Exam Component 


Untrained 


Trained 


t 


Significance 


Composite 


374 


497 


21.45 


< .001 


MCQ 


341 


503 


22.52 


< .001 


PMP Data Gathering 


377 


493 


16.45 


< .001 


PMP Management Aherapy 


403 


495 


12. 14 


< .001 



A third validation study was conducted in order to estimate the concurrent validity of the Certifying 
Examination. This study u'as desired to investigate the relationship between performance on the 
various components of the Certifying Examination and ratings of clinical competence as provided 
by program faculty. A clinical competence rating form was developed consisting of 40 statements 
each describing different aspects of tlic competency of physician's assistants. These statements 
described behavior related to: (a) history-taking, (b) physical examination, (c) laboratory tests 
and diagnostic procedures, (d) management/treatment, (e) medical records, and (f) interpersonal 
relations. 
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Individuals who had registered for the 197? Certifying Examination and who were still in 
training or had recently completed training were identified. A clinical competence rating 
form for each of these individuals was mailed to the appropriate program director who was 
asked to identify a member of his faculty who was familiar with the clinical performance 
of each examination candidate^ The faculty rater was asked to read each statement contained 
in the rating form and decide at wliat level of competence the candidate performed. A scale 
consisting of five intervals was used with the end points labeled "minimum competence" and 
^bptimum competence.*' 

Of the 179 rating scales mailed to program directors, 154 completed forms were returned, 
representing an 86 per cent return rate. Of tins number, four were deleted from the sample 
since tlie registrants did not take the Physician's Assistant Certifying Examination. Thus, the 
subjects of this study were 150 registrants for whom a set of examination scores and ratings 
of clinical competence were available. 

Since only a sampling of the total registrant pool was Involved in the study, a preliminary 
analysis was performed to determine whether or not these subjects were representative 
of the registrant population. Accordingly, significance tests were performed between the 
study subjects and the remainder of the examinee population in terms of the distribution of 
performance on each major examination component. These comparisons consisted of t-tests 
between tlie means and F-tests between the variances of the two examinee groups. In all cases, 
these statistics were not significant, mdicating no significant differences between the groups in 
terms of tiieir perforniance on the major components of the examination. These results suggest 
that the st\idy sample was representative of the population of examinees in terms of examination 
performance. 

A factor analysis of the items contained on the rating scale was performed in order to identify 
clusters of related items and the dimensions of clinical competence each cluster appeared to be 
measuring. The factor analysis yielded the following three dimensions: (1) Data Gathering and 
Recording, (2) Interpersonal Skills, and (3) Clinical Judgment. The Data Gatliering and Recording 
factor consisted of rating items assessing competence in taking histories, performing physical 
examinations, and recording patient data. The Interpersonal Skills factor contained rating items 
pertinent to interactions bet^veen the physician's assistant and the patient, the patient's family, 
and other members of the health team. The Clinical Judgment factor consisted of items pertinent 
to various aspects of patient management. Analysis of the examination and rating scale data was 
performed by correlating an individual's clinical competence rating on each of the three factors 
with the various examination components. In order to provide a more detailed analysis of tlie 
examination itself, items on the multiple-choice question portion were classified into one of the 
following four categories; (1) identification and classification of physical findings, (2) patient 
management, (3) knowledge of clinical procedures, and (4) Interpersonal skills. 

Pearson product-moment correlation coefficients were computed to determine the direction and 
magnitude of the simple correlations between examination components and rating factors. In 
those instances where an examination component correlated significantly with more than one 
rating factor, step-wise multiple regression analyses were performed to identify the nature and 
magnitude of these complex relationships. Evidence for the construct validity of the Certifying 
Examination would be provided by the extent to which significant positive correlations were 
observed between those examination components and rating factors where it would be logical to 
expect such a relationship. The results of these analyses are summarized in Tables 7 and 8. 

Candidate performance on multiple-choice questions dealing with the identification and classifica- 
tion of physical findings correlated significantly with ratings of Data Gathering and Recording 
skills and Clinical Judgment. One would not expect performance on tliis examination component 
to correlate with ratings of Interpersonal Skills, and, in fact, such a relationship was not 
observed. Performance on multiple-choice questions related to patient management correlated 
significantly with ratings of Clinical Judgment, but not with ratings of Data Gathering and 
Recording or Interpersonal Skills, a pattern of correlations which is consistent with the logical 
constructs of these examination and rating scale components. 
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Similarly for patient counseling and instruction, examination performance on those multiple- 
choice questions correlated sit^^ificantly witii ratings of Interpersonal Skills and Clinical 
Judgment. One would not expoct a significant relationship to ratings of Data Gathering and 
Recording skills, and none was observed. Performance on the PMP data gathering section 
(which included history-taking, physical examination, and selection of laboratory tests) 
correlated significantly with ratings of Data Gathering and Recording skills and Clinical 
Judgment. Since Interpersonal Skills were not measured on this component of the examination, 
one would not expect to find a si[;nificant relationship with ratings of Interpersonal Skills, and 
this was observed to be tlie case. Of tlie four examination components thus far described, all 
were found to correlate significantly with Uie rating factors to which thoy bore the most logical 
relationship and not witJi rating factors that did not seem pertinent to the nature of tlie examina- 
tion component. To this extent, Uie pattern of significant correlation provides evidence for 
the convergent and discriminant validity of tJie Certifying Examination. 

As will be noted in Table 7, multiple- choice questions related to the knowledge of clinical 
procedures did not correlate significantly witJi any of tlie three rating factors. This, too, 
is consistent with evidence for tiie discriminant validity of the Certifying Examination since 
none of tlie rating factors dealt witii a candidate's level of competence in performing clinical 
procedures. Thus, the absence of significant correlation is consistent with the logical constructs 
of the examination components and rating factors. The one examination component for which 
evidence of discriminant validity was not observed was that component of the patient management 
problems that dealt with management and tlierapy decisions. One would have expected to observe 
a significant correlation between this examination component and ratings of clinical judgment. 
Although the magnitude of tlie correlation coefficient between PMP management/therapy and 
ratings of Clinical Judgment was greater than that observed for the other two rating factors, 
it did not reach statistical significance. A further analysis of the statements contained on the 
Clinical Judgment rating factor indicated that behavior related to the selection and sequencing 
of appropriate diagnostic tests and procedures made up a large number of the items included 
on tins rating factor. However, the examination component labeled PMP management /therapy 
did not include those items on the patient management problems related to the selection and 
Interpretation of diagnostic tests and procedures. Instead, these items were included as part 
of the PMP data gathering component. It is possible, therefore, that the lack o! significant 
correlation was attributable to the incorrect classification of patient management problem 
items related to the selection of laboratory tests. Such a reclassification of these items has 
been performed for the 19 74 Certifying Examination, and this analysis will again be performed 
in order to provide further data concerning the discriminant validity of this examination 
component. Table 8 summarizes the multiple correlation coefficients obtained for three of 
the six examination components. The magnitude of the multiple correlations is greater than 
that for the simple correlations because more than one rating factor is being used to predict 
performance on a single examination component. 
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Table 7 

Siinple Correlations Between Rating Factors 
and Components of Examination Performance (N=124) 



Examination Component 



Rating Factor 
Data Gathering Interpersonal 
and Recording Skills 



Clinical 



Identification and Classification 
of Physical Findings 


.18* 


-.02 


.25** 


Patient Management 


' .08 


.11 


.21 * 


Patient Counseling and Instruction 


. 13 


.23 * 


.23* 


Knowledge of Clinical Procedures *** 


.04 


.04 


.16 


PMP Data Gathering 


.29 *♦ 


.12 


.26** 


PMP Management Aherapy 


.06 


.07 


.11 



♦ Significant beyond the 05 level 
♦* Significant beyond tlie .01 level 



Rating scale did not contain items that permitted faculty to indicate registrant's level of 
competence in performing clinical procedures. 



Table 8 

Complex Correlations Between Rating Factors 
and Components of Examination Performance 



Examination Component 



Rating Factors 



Multiple 
Correlation ♦ 



Identification and Classification 
of Physical Findings 

Patient Counseling and 
Instruction 

PMP Data Gathering 



Di».ta Gathering 
Clinical Judgment 

Interpersonal Skills 
Clinical Judgment 

Data Gathering 
Clinical Judgment 



,29 



.33 



,37 



All correlation coefficients are significant beyond the .05 level 
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While the pattern of significant correlation coefficients between examination performance and 
clinical competence ratings provided e/idence for the convergent and discriminant validity of 
the Certifying Examination, the magnitude of these correlation coefficients was modest< An 
analysis of the rating scale itself suggested tliat several modifications could be made in the 
format which might enhance the sensitivity of this measuring instrument. (One of the difficulties 
that has long plagued ratmg scales has been their inability to distmguish different levels of 
competence within the same individual along different rating dimensions,) For this reason, 
significant modifications were made in tiie rating scale and a replication of Uds study carried 
out in relation to the 1974 Certifying Examination. 

The following modifications were made in the rating scale used in the replication, 

1. the number df competency statements to be rated was increased from 40 tu 52; 

2. raters were asked to indicate the degree of confidence they had in the rating 
given to each physician's assistant for each competency statement; 

3. raters were offered the option of indicating that for a given competency 
statement they did not feel able to judge the physician's assistant. 

The results of analyses between clinical competence ratings and examination scores did not 
reveal a marked increase in the convergent or discriminant validity of tlie Certifying Examination. 
In addition, the factor analysis of rating scale items did not yield markedly different factors from 
those that had been observed in the first study . However, it was found that confidence in the ratings 
provided differed significantly among the three groups of individuals (program administrators, 
program faculty, and employers) who provided ratings on physician's assistants. In general, 
program administrators were significantly less confident in their ratings of clinical competence 
than were program faculty or employers. This finding suggests that certain types of individuiils 
should not be asked to provide ratings of competence where they have limited observations and/or 
experience on which to base their ratings. 

(A detailed report of the 1973 study was published in the Journal of Medical Education , March 1976.) 
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EXPANDED ELIGIBILITY - 1974 CERTIFYING EXAMINATION 

In addition to graduation from an approved educational program, eligibility for the 1974 
Certifying Examination was expanded to include individuals qualifying on the basis of v/ork 
experience. Individuals wishing to so qualify were required to have a high school diploma or 
an equivalency certificate and four years of medical clinical experience in primary care as a 
physician*s assistant or nurse practitioner since January 1, 1970. The work experience out- 
lined on each application form was verified by contacting the physicians whose names and 
addresses were provided. Each physician was required to provide a detailed description of 
the health care functions performed by the applicant as well as an estimate of the frequency 
with which each was performed. These details of an applicant's employment history were 
then evaluated in relation to specific criteria that had been established in advance by the 
Eligibility Committee. In instances where an applicant was not clearly eligible or ineligible, 
the application form and supporting documents were reviewed individually by members of 
the Eligibility Committee. 

Announcements of the eligibility requirements were placed in the major primary care medical 
journals and newsletters. As a result, 550 completed application forms were received (rem 
individuals wishing to qualify on the basis of work experience. Of this num.bcr, 150 met the 
established eligibility requirements and 116 actually sat for the Certifying ExaminaLion. The 
most frequently encountered factors in failing to meet eligibility requirements involved less 
than the required four years of clinical experience and health care functions that did not include 
making patient management and disposition decisions* 

1974 PHYSICIAN'S ASSISTANT CERTIFYING EXAMINATION 

The written component of the 1974 Certifying Examination followed the same format as was 
used in 1973: that is, a multiple-choice examination designed to assess the candidate's knowledge 
and skill in applying knowledge related to clinical material presented in printed and pictorial form, 
and patient management problems involving simulated clinical cases in adult and pediatric medicine 
designed to assess the candidate's skill in gathering pertinent information about patients and in 
making appropriate management decisions. The number of items included on both portions of 
the written examination was increased on the 1974 Examination. In addition, multiple-choice 
questions using pictorial material involved the identification and interpretation of physical 
findings instead of the identification and classification of findings as had been tlie case in tlie 
previous year. 

In addition to the written component of the examination, each candidate underwent assessment 
of hisAer physical examination skills using the standardized behavioral checklists that had been 
developed as part of a research and development project. This assessment consisted of an 
evaluation of the candidate' s proficiency in performing five components of the physical 
examination (i.e., heart, lungs, eyes, abdomen, and neurologic) . Test centers were 
established on tlie two days following the written examination and candidates were given 
Individual appointments for asse.ssment. The candidate performed three examinations on 
one patient while being observed by a physician examiner, and performed the remaining two 
examinations on another patient while being observed by another physician examiner. The 
total assessment time for each candidate was approximately forty minutes. At the completion 
of each examination, the candidate was asked to describe any abnormalities found . However, 
due to the variability in abnormal findings among patients and the limited sample of abnormal 
findings that was available, this portion of the assessment procedure was not used for scoring 
purposes. (A detailed report concerning the development and field testing of these checklists 
is provided in the Project Report, 1973-74.) 
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Physician examiners were appointed by program directors at the various test centers Prior 
to the examination, three regional orientation meetings were held witli examiners in order to 
acquaint them witli the assessment objectives, procedures, and guidelines for patient selection. 
In addition, proctors were appointed for these special test centers to ensure the adequacy of tlie 
logistical arrangements and to maintain Uie security of the assessment procedure itself. Since 
written reports from the chief proctors and physician examiners did not disclose any problems 
that would jeopardize the integrity of this assessment procedure, tlie physical examination 
assessment component of the Certifymg Examination was used for actual scoring purposes Each 
of the five physical examination components was weighted equally, and performance on this component 
of the examination accounted for tw»enty-five per cent of a candidate's total score- 

STATISTICAL ANALYSIS - 1974 CERTIFYING EXAMINATION 

The 1974 Certifying Examination was administered on December 11-13 to 1303 candidates in 
49 test centers. The statistical properties of the 1974 examination program closely paralleled 
those encountered in 1973. Altlioagh the number of candidates still enrolled in educational 
programs increased to twenty-three per cent in 1974 as compared to eleven per cent in 1973, 
the typical examinee continued to be an individual who had already completed a formal educational 
program and acquired up to t^vo years of postgraduate clinical experience. Seventy -five per cent 
of the formally trained examinefts received their training in programs from 13 to 24 months in 
length, and ninety-two per cent had been involved in health care delivery piior to being trained 
as a physician's assistant or nurse practitioner. Of this number, eighty-four per cent had been 
involved in pcttient contact health care delivery. Of \}\ose examinees with prior experience in 
health care delivery, fifty -eight per cent had more than four years of experience. These data 
are sunimirized in Table 9 on the following page. 
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Table 9 

Physician's Assistant Certifying Examination 
Description of Examinee Populations 

19 73 19 74 

I. Percentages of total respondents: 

A. Current status 

Currently in educational program • 11% 23% 

Currently employed or graduated 89 77 

B. Type of training 

Physician's assistant^ssociate 62 71 

Medex 29 " 15 

Nurse 9 5 

Informally trained — 9 

C. Amount of experience as a P. A. 

None 19 25 

0-2 years 72 62 

More than 2 years 9 13 

n. Percentage of respondents with formal training ) 
(informally trained candidates not included) 

A. Length of educational program 

4-12 months 22 13 

13-24 months 65 75 

25-36 months 10 7 

More than 36 months 3 5 

B. Experience in health care delivery prior to 
educational program 

Yes 91 92 

No 9 8 

1. Type of experience 

Technical 11 12 

Patient contact ^ . 86 84 

Other 3 4 

2. Length of experience 

0- 1 year 4 4 

1- 2 years 10 11 

2- 4 years 30 27 
More than 4 years 56 58 
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Statistical analysis of the 1974 examination indicated that the average difficulty level and 
reliabilities for the various components of the examination were comparable to statistics 
derived from tlie 1973 examination. The one exception is the average difficulty level of the 
multiple-choice questions which indicated that those developed for the 1974 examination were 
somewhat more difficult tlian the MCQ's developed for the 1973 examination. (In 1973, the 
average difficulty level equaled .64; in 1974, tliis figure was .57.) The composite reliability 
for all examination components increased from .89 in 1973 to .93 in 1974. This modest increase 
in reliability is attributable to the larger number of items contained on the 1974 examination. 
Comparative statistics for the first two Certifying Examinations are provided in Table 10. 

An analysis of the inter-correlations amoug the various examination components yield 
correlation coefficients ranging from , 13 to .53. The correlations among the portions of the 
written examination ranged from .37 to .53, and closely resembled those encountered on the 
1973 examination program. These modest correlations suggest Uiat the various portions of 
the written examination are measuring different aspects of competence. As might be anticipated, 
the correlation between tlie written portions and tlie physical examination assessment portion 
of the Certifying Examination were lower, ranging from • 13 to .28. (These inter-correlations 
are summarized in Table 11.) Given the magnitude of these inter-correlations, it is evident 
that one cannot predict a candidate's physical examination skills by knowing his score on the 
written portion of the examination. Thus, it appears reasonable to conclude that the performance 
component of the examination was contributing significant information concerning the competence 
of candidates that could not be inferred from performance on the written portions of the examination. 
Moreover, the wide variation in scores on the physical examination assessment component indicated 
a wide range of proficiency among the 1303 candidates who sat for the 1974 examination'. This 
range of scores was greater than that observed for the written component of the examination. 

Table 10 

Comparative Examination Statistics (1973-1974) 



Exam Component 



Number of 
Items 



Average 
Difficulty 



Reliability 



Composite 
Reliability 





1973 


1974 


1973 


1974 


1973 


1974 


1973 


1974 


MCQ 


147 


216 


.64 


.57 


,83 


.89" 






PMP - Data 
Gathering 


422 


554 


. 79 


. 75 


.80 


.89 


- .89 


- .93 


PMP - Management/ 
Therapy 


37 


284* 


.82 


.78 


. 75 


.76_ 






Physical Examination 




212 




.67 




.91 







Skill Assessment 



♦ The increase in the number of items reflects the fact that the selection of laboratory studies was 
included in the management sub-score rather than in the data gathering score as had been tlie case 
in 1973. 
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Table 11 



INTER-CORRELATIONS AMONG SUB-SCORES 
1974 CERTIFYING EXAMINATION 



PMP (Mgmt. Aherapy) PMP (Data/Cath.) 



Physical Exam 
Assessment 



MCQ 



.53 



.41 



.28 



PMP (Mgmt. /Therapy) 



.37 



.12 



PMP (DataXJath.) 



.13 



In addition to the above analyses, a comparison was made of the performance of informally and 
formally trained candidates on all components of the Certifying Examination, In all instances, 
candidates who had not graduated from formal educational programs scored significantly lower 
than did candidates who had been trained in formal educational programs. With respect to 
examinees who had been traiiied in formal educational programs, candidates from physician's 
assistant and nurse practitioner programs scored equally well on the written portions of the 
examination. However, candidates from Medex programs scored significantly lower on the ' 
written examination when compared witli the two other groups of formally trained examinees. 

With respect to the physical examination skill assessment, candidates from physician's assistant 
programs scored significantly higher than candidates from Medex or nurse practitioner programs. 
There was no significant difference in the performance of Medex and nurse practitioner candidates 
on the physical examination skill assessment. In interpreting the findings concerning nurse 
practitioners, it should be remembered that the number of nurse practitioners sitting for this 
Certifying Examination was not only very small (N=:69), but was probably not representative of 
nurse practitioners in general. The number of nurse practitioner candidates is probably some- 
what less than 10 per cent of the total number of nurse practitioners trained in any given year. 

As the 1974 Certifying Examination incorporated a new evaluation methodology for the assessment 
of physical examination skills, it was hoped that research on audiovisual simulation would result 
in the development of yet additional new evaluation methodologies for the assessment of another 
Important area of physician^ s assistant competence interpersonal skills. 

Setting Examination Standards 

The method that has become most wjdely used for setting pass-fail levels on standardized 
examinations involves the use of norm-referenced procedures. When these procedures 
are employed, a pass-fail level is determined on the basis of how well examinees perform 
in relation to one another. Witliin the last few years, a number of articles have appeared 
in the literature describing different procedure^ for setting examination standards. These 
procedures have been described as criterion-referenced and differ from norm-referenced 
procedures in that the pass-fail level is established as a function of a predetermined standard 
rather than a5 a function of the relative performance of examinees in relation to one another. 
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In order to investigate the nature and stability of tJiese criterion-referenced procedures, t"^vo 
were selected and used experimentally by test conimitlces for the 1973 Physician's Assistant 
Certifying Examination. The purposes of this study were to determine: 

L whetlier the pass -fail levels for comparable samples of test content would vary 
depending upon tlie standard setting procedure used to set tins level; 

2. whether for each of two standard setting procedures, the pass-fail level for sample 
test content would vary depending upon the group of judges whose decisions were used 
to set tlie standard; 

3. whether the average of judgments made by individuals within each group would differ 
significantly from the group judgments arrived at by consensus.. 

The two standard setting procedures selected were chosen because both were designed to set 
an examination standard in relation to a 'himimum acceptable level of proficiency, "and were 
amenable to use with individual or group jud^^ments. Although the two procedures differed in 
terms of Uie specific techniques used to develop the pass-fail level, both procedures required 
groups of judges to review each test item and make a decision concerning how well a minimally 
qualified candidate should perform in relation to it. Judgments concerning individual test items 
were then combined in such a way as to provide an overall pass-fail level. 

Two groups of judges were selected at random from among the various test committees. Each 
group utilized both standard setting procedures. On the basis of the analyses performed, the 
following conclusions were reached: 

1. Different groups of judges using the same criterion-referenced procedure in relation 
to the same examination content did set similar overall pass-fail levels. 

2. When tlie magnitude of agreement between groups of judges using the same criterion- 
referenced procedure was estimated on an item-by-item basis, product-moment 
correlation coefficients of .40 - .45 were obtained. This indicates a modest positive 
correlation between groups of judges with respect to the judgments they made about 
individual test items when using the same criterion-referenced procedure. 

3. The averages of judgments concerning examination standards made by individuals 
within each group did not differ significantly from the consensus judgments of the 
group as a whole . 

4. The two criterion-referenced procedures used in this study did, however, yield 
significantly different pass-fail levels for comparable samples of test content. 

This study was replicated using the 1974 Certifying Examination and different groups of judges. 
The purpose of this replication was to validate the conclusions reached in the first study The 
preliminary statistical analysis indicates that the two criterion-referenced procedures continue 
to result m different pass-fail levels In contrast to the previous study, however, it appears that 
the two groups of judges set different pass -fail levels even when using the same standard setting 
procedure. 

Although it IS premature to formulate any definitive conclusions regarding the nature of criterion- 
referenced procedures for setting examination standards, certainly the results of the second study 
raise a question concerning the stability of either procedure in setting a consistent pass-fail level 
for a given sample of examination content. While these findings probably do not differ from those 
that would be observed with use of norm- referenced standard setting procedures, they do challenge 
the assumption that has been made by many tliat criterion- referenced procedures constitute a more 
objective and stable means of setting examination standards. 
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It is i^rhaps reasonable to think that the pasft-fail level set by whatever standard setting 
procedure will vary significantly as a function of tlie individuals who are selected to set 
tliat standard. For this reason, it is conceivable that decisions to use one standard setting 
procedure as opposed to another will be made largely on philosophical grounds raUier than 
upon statistical data to support one approach over another, A further problem which must 
be faced is that criterion-referenced procedures are applicable only to multiple-choice and 
matching type test items, and cannot be used with items on patient management problems 
which constitute more than half of the Physician* s Assistant Certifying Examination. 
(A report of this research study was published in Educational and Psychological Measurement ^ 
Spring 1976,) 

The following sections of this report contain a detailed description of activities carried out 
during Phase II of this project including: (a) a study of audiovisual simulation techniques 
for assessing inteiT}ersonal skills; (b) a validation study of the 1974 Certifying Examination 
involving medical record audits; and (c) an analysis of the 1975 Certifying Examination. 
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AUDIOVISUAL SIMULATION STUDY 



Introduction 

Whcin historians and social scientists attempt to synthesize the accomplishments 
of the first three quarters of the twentieth century and characterize the 
evolution of civilization as manifest during that period, it will no doubt be 
man's scientific and technologic advances which will occupy the major part of 
their narrative. As literature, philosophy, and the arts have constituted the 
humanistic legacy of man's achievements in the past, this century will no doubt 
be viewed as a watershed in man * s scientific heritage. To the extent that 
science has preoccupied our creative and pragmatic energies in this century, 
the humanistic dimensions and their potential contribution to our quality of 
lif-ti have been overshadowed. 

In the health professions, this has resulted in a heightened capability 
to comprehend and master the complex biophysical processes which determine 
health status. In large measure, however, health professionals have lost sight 
of and insight into the psychosocial components of man and his environment and 
the implications of these components for health. Armed with the powerful 
weapons of science, health professionals have neglected the humanistic aspects 
of problem-solving and have not fully developed the skills necessary for 
effective human interface. 

Within the last fifteen years, studies whose focus has been the definition 
of professional competence have begun to highlight a nunber of dimensions which 
have renewed interest in what might be considered the humanistic components of 
competence. As an example, in I960, the American Institutes for Research 
conducted a critical incidents study of intern and resident performance for the 
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National Board of Medical Examiners. The purpose of this study was to identify 
critical incidents of physician performance which were felt to have a positive 
or negative impact on the quality of health care. Interwoven among the over 
3,000 critical Incidents which were gathered from this study are examples of 
behavior which might be described as interpersonal and communication skills. 
In other studies of clinical competence using somewhat different methodologies, 
medical specialty boards and societies have also included these dimensions as 
important components of physician competence. It is perhaps not surprising, 
therefore, that definitions of clinical competence for the primary care 
physician's assistant have also included interpersonal and communication skills. 

Since these skills were felt to be important competency components for the 
primary care physician's assistant, a project was undertaken to develop a 
detailed definition of these skills and to design and validate several evaluation 
methodologies which could assess these components of competence for purposes of 
certification in the profession. 

Criterion Development and Measurement Issues 

It was felt that the assessment of interpersonal and communication skills 
should be undertaken within the context of patient interviewing and counseling 
activities. The first step in this process was to identify performance 
criteria that could serve as the basis for evaluating the behavior of physician's 
assistants. In order to identify appropriate performance criteria, a project 
committee was appointed. The committee was composed of individuals who were 
active in teaching and evaluating interperi^onal and communication skills within 
physician's assistant training programs. Membership was inter-disciplinary 
and included representaion from medicine , surgery, pediatrics, psychiatry, 
clinical psychology, and nursing. 
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Performance criteria associ,ated with patient interviewing and counseling 
were specified as presented in Table 1 below. 



Table 1 

Performance Criteria for Patient Interviewing 
and Counseling 



I, Content of Question s or Statements 

A. Uses questions or statements that will provide pertinent data about 
the patient. 

B. Does not use questions or statements which will frighten, anger, insult, 
or embarrass the patient. 

C. Questions patient about inconsistencies. 

D. Uses language anpropriate to the educational level and background of 
the patient. 

E. Avoids use of rriedical "jargon.'^ 

F. Collects sufficient data before terminating the interview, 
I I . Format of Question s 

A. Selects question formats which will yield the greatest amount of data 
with the greatest accuracy. (These will generally be open-ended 
or reflective questions.) 
III. Cue Sensitivi ty 

A. Follows up pertinent verbal and non-verbal cues provided by patient. 

B. Encourages patient v) express pertinent information and feelings. 
IV. Patient Ra pport 

A. Appropriately answers patient questions (giving appropriate information 
without frightening, angering, or embarrassing the patient). 
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B. Provides patient with appropriate information (in terms of accuracy 
and amount). 

C. Deals appropriately with different patient personalities. 

D. Gives reassurance and support where appropriate. 

E. Appropriately expresses understanding and/or empathy. 

F. Maintains a non--judgmental , respectful attitude toward the patient. 

G. Maintains eye contact wLth the patient. 

H. Acknowledges patient comments and concerns. 

I. Uses gestures, facial expressions and posture which convey attention 
and interest. 

J. Does not interrupt the patient. 

K. Terminates interview in an appropriate manner. 

(1) Asks if there*s any additional information patient wants to provide. 

(2) States immediate disposition of what is to happen next. 



In reviewing the performance criteria specified in the above and considering 
the nature of the activites to which these criteria were related, it was felt 
that three dimensions were crucial to these activities and should be captured 
to the extent possible in the design of evaluation methodologies. Since the 
course of a patient interviewing or counseling session ^,s determined in large 
measure by the nature of the verbal and non-verbal cues presented by the patient, 
it was felt that audiovisual stimuli would be an important component of any 
evaluation methodology. In addition, since it is probable that considerable 
variation will exist among health professionals in terms of their approach to a 
specific interviewing or counseling session, it was felt that capacity for 
branching logic and multiple pathways would be another dimension that should 
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be included in an evaluation approach. Finally, the need to provide audiovisual 
stimuli and branching logic led to the identification of interactive capability 
as a third relevant dimension in the evaluation process. 

Associated with any evaluation enterprise is the need to identify not onlv 
relevant performance criteria and evaluation dimensions but also the need to 
identify measurement problems and aspects of economic and logistic feasibility 
that will influence the selection and design of appropriate evaluation methodologies 

In relation to measurement problems, two seemed particularly relevant: 
1) standardization of the content of an evaluation technique and 2) development 
of scoring strategy that numerically reflects the quality of individual 
performance criteria. ^ 

To the extent that observation of live performance is used as one evaluation 
approach, the problem of standardization is a particularly difficult one. 
Moreover, the observation of live performance results in generally higher costs 
and more complex logistics associated with its administration. Unfortunately, 
our experience to date with observation of live performance suggests that while 
the measurement and logistic problems can be overcome, the cost of doing so 
raises major questions about the ultimate feasibility of utilizing observation 
of live formance as an ongoing mechanism for evaluating professional competence. 
However > if these problems can be brought within acceptable levels of tolerance, 
it remains that the evaluation of actual performance yields more accurate 
assessment of performance than any indirect ass^^ssment technique be it paper and 
pencil, audiovisual simulation, or computer simulation. 
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Thus, the challenge that faces agencies with responsibility for evaluating 
professional competence is the development of indirect evaluation methodologies 
that are capable of predicting with reasonable a ^uracy the performance of 
health professionals as it would be observed in an actual clinical setting. 
In this context, the measurement of live performance can be used as a criterion 
measure against which to assess the validity of indirect evaluation methodologies. 
To the extent that one or more of these methodologies yields significant positive 
correlations with actual performance, their validity can be judged. 

Evaluation Methodologies 

It was concluded that the basic research design for this study would 
utilize the observation of live performance in patient interviewing and counseling 
sessions as the criterion measure against which to assess the validity of 
several evaluation approaches involving three types of simulations. 

Several studies in the literature have reported the use of interaction 
observation forms for evaluating patient interviewing and counseling 

skills. (1-6) These observation foms have contained specified categories of behavior 
which were to be observed and recorded by a trained observer as a physician- 
patient interaction was under way. Since studies involving a limited number 
of behavioral criteria had demonstrated that acceptable reliability coefficients 
could be obtained using this evaluation approach, it was felt that a similai 
approach should be developed and reliability studies conducted in order to 
develop the criterion measures of performance for this study. 

However, two aspects of this study differed significantly from previous 
efforts. First, the number of behavioral criteria that needed to be included 
on the observation fonn was greater than the number of criteria that had been 
utilized in previous studies. Thus, a significant question was whether trained 
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observers could learn a large number of behavioral categories, quickly classify 
observed behavior into one or more categories, and record the decisions accurately 
on the observation form without jeopardizing the accuracy and reliability of 
their observations. Second, although previous studies had focused on the 
description of behavior during interviewing and counseling sessions, none of 
these studies had atr.erapted to develop a scoring strategy which could describe 
the quality or appropriateness of the behavior that had been observed. Thus, 
the problem of placing a value judgment on observed behavior had to be faced 
in the conceptualization of a logical and reasonable scoring strategy. 

Three indirect evaluation approaches were included in this study. Each 
approach differed from the others in the extent to which relevant dime ^ s 
(i.e., audiovisual stimuli, branching logic, and interactive capability) could 
be incorporated. The three evaluation approaches included: 

a. problem- centered mul tip le- choice questions 

b, branching, paper-pencil simulations 
Cv interactive audiovisual simulations* 

All evaluation approaches involved the presentation of problem situations 
in which the subject was required to make decisions concerning the question 
or comment to the patient which would be most effective in dealing with the 
specific problem situation. However, the multiple-choice questions did not 
provide for audiovisual stimuli j interaction, or branching capabilities. 
A sample of the problem-centered multiple-choice questions is provided in 
Table 2. 
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CASE B (Questions 5-11): 



Table 2 

Sample Multiple-Choice Questions 



Kathy D., a 14-year-oid eighth grader, comes to the office because of two rnissed menstrual 
periods. During the history you learn that Kathy 's menarche occurred at age 12 and for the 
past year menstruaticn has been regular every 28 days with the exception of the past two months. 
You further learn that Kath> has been dieting rigidly over the past 4 months in an effort to lose 
15 pounds. 

From your viewpoint she now looks too tliia, her clothes hang, her cheekbones show, her hair 
and eyes lack luster, and she is less vivacious than when you saw her a year ago. She is 5' 4" 
and weighed 110 pounds before her diet. 

She tells you that she has lost 12 pounds and only has three to go. She says, 'This diet has been 
the hardest thing I've ever done, but look at me. . .Don't I look terrific?" 

5. The best immediate response is 

(A) "It must have been hard to lose all that weight in just a few months. " 

(B) "You really look too thin, Kathy." 

(C) ".. .terrific?" 

(D) "What made you go on the diet in the first place?" 



6. When you ask Kathy how she feels in general, she tells you that lately she has been tired more 
than usual. She also says that she notices the cold more than in the past, and finds herself 
wearing a sweater at times that she would normally be comfortable with just a dress or pants 
and a shirt. She says, "This being tired all the time is such a drag. I get more than enough 
sleep, but Vm still tired. How come?" The best response would be 

(A) "I expect one reason is that you've cut down on a good source of energy - food " 

(B) "It may be that your sleep is not restful." 

(C) "It's not unusual for adolescents to feel tired much of the time. Your body is 

actively maturing, which is hard work." 

(D) "Before drawing any conclusions, I'd like to examine you and do some tests," 



7. When you ask Kathy how her parents feel about her dieting, she tells you that it's given 
them one more thing to harp on. She says, "You know my parents have been all over me 
shice I entered junior high. If it's not my grades, it*s my language, my clothes, my messy 
bedroom, my laziness, and now U's my eating or lack of it* I wish they'd bug off." The 
best response is 

(A) "1 imagine your parents are concerned about your welfare." 

(B) "It must be difficult to live in such a negative atmosphere." 

(C) "Why don't you bring your parents in with you for a conference?" 

(D) "It sounds as though your life changed after you entered junior high." 



Kathy continues talkmg about her parents and says, "Actually, I'm here because my mother 
made me come. She got all upset when she found out I missed a couple of periods. She made 
such an issue out of it. You'd think I was pregnant. What's the big deal about a couple of missed 
periods?" The best response is 

(A) "Did you mother have reason to think you might be pregnant?" 

(B) "What does a missed period mean to you?" 

(C) "Your mother is probably worried that your dieting resulted in the missed periods. " 

(D) "Sometimes missed periods can be a warning signal of problems other than pregnancy. 
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The branching paper-pencil simulations were designed so that each 
simulation began with a case description of the patient and offered a variety 
of questions and statements from which the subject could select the one he 
felt was most appropriate for that point in the simulated ihteraction* As 
each question or comment was selected, the patient's response could be read 
by using a latent image pen to develop the invisible print. At the end of 
each patient's response, instructions were provided for turning to the next 
series of options and selecting the next question or statement* Since each 
simulation involved branching logic, subjects were able to follow different 
pathways of interaction with each Simulated patient. Moreover, the patient's 
response to each question or statement provided the basis from which a thread 
of continuity could be established in the interaction. Thus, this evaluation 
approach was able to capture two of the three relevant dimensions of the 
intenriewing and counseling process: branching logic and interactive capability. 
Appendix A contains a sample paper-pencil simulation with all patient responses 
exposed for illustrative purposes. 

The use of interactive audiovisual simulations necessitated the conceptualiza- 
tion and development of an innovative system which would be capable of providing 
random access to audio and visual stimuli as well as interaction with the 
simulated patient by means of a branching logic and videotape program, A 
prototype Interactis^e Audiovisual Simulation System had been developed in 
connection with an earlier project and was incorporated into the design of this 
study . 

The unique aspects of this Interactive Audiovisual Simulation System 
involve the interface between random access slide projection and random access 
videotape display, flexible branching logic, rapid and accurate search and 
retrieval, and a hard copy printout of selections made, 
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in using this "system, the subject sits in front of a display console and 
interacts with a patient who has been filmed on videotape according to 
a branching logic script. The subject advances through each nodal point of 
the branching simulation by means of selecting from a series of options the 
question or statement he believes appropriate for that moment in the interaction 
with the patient. The code number of the option is entered into the system 
which then retrieves and plays the videotape segment containing the patient *s 
response to the subject* s selection. Based upon the audiovisual cues presented, 
the subject can then decide whether to pursue the same line of questioning or 
to begin a new aspect of inquiry. Some pathways lead to the successful 
resolution of the interviewing or counseling session, while others result in 
a premature termination of the session because an ineffective or inappropriate 
approach to the patient has been selected or because the interviewer chooses 
to terminate the interaction at an earlier point. 

In order to develop interaction observation forms for assessing performance 
in a live interview and counseling session, the performance criteria that had 
been specified by the project committee were reviewed and categories of 
appropriate and inappropriate beha\/ior consistent with each performance criterion 
were identified. As a result, 28 categories of behavior were developed. These 
categories are listed in Table 3. 
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Table 3 
Behavioral Categories 
Interaction Observation Form 



Question Content /Format 



1. 


Asks 


non-case pertinent, open-ended question 




2. 


Asks 


case pertinent, open-ended question 




3. 


Asks 


non-case pertinent, close-ended question 




4. 


Asks 


case pertinent, close-ended question 




5. 


Asks 


case pertinent. Inappropriately leading question 




6, 


Reflects patient comment or question as a probe 




7, 


Asks 


repetitious question (l,e,, the same question asked multiple 


times) 


8, 


Uses 


continuous questioning (l.e,, asks more than one question at 


the 




same 


time) 





Reactions to Patient Behaviors 

9, Inappropriately Interrupts patient /abruptly shifts focus of Interaction 

10, Encourages patient to continue talking 

11, Uses medical jargon 

12, Follows up pertinent verbal or non-verbal cues (new information) 

13, Fails to follow-up pertinent verbal or non-verbal cue (new information) 

14, Repeats own prior question/comment verbatim 

15, Clarifies own prior question /comment 

16, Appropriately summarizes/synthesizes patient comments 

17, Inappropriately summarizes/synthesizes patient comments 

continued 
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Provision of Support /Information 

18. Expresses understanding of what patient has said 

19. Responds to patient comments with inappropriate affect (e.g., disinterest, 
judgment, disapproval) 

20. Reassures patient appropriately 

21. Reassures patient inappropriately 

22. Answers patient^s questions directly 

23. Answers patient *s questions evasively 

24. Gives advice/instructions appropriately 

25. Gives advice /instruct ions inappropriately 

) 

26. Provides appropriate information 

27. Provides inappropriate information 

28. Expresses empathy 



The above behavioral categories were then formulated into an observation form 
as illustrated in Appendix B. The observation form was designed so that a 
trained observer could record the behaviors which were observed in the 
sequence in which they took place. In instances where the term "appropriate" 
appears in relation to a behavioral category, it was felt that the appropriateness 
or inappropriateness of the behavior would be a function of the specific patient 
and his or her problems, personality characteristics, and purpose of the 
interview or counseling session. 

Developm ent of Patient Protocols 

In order to develop the clinical cases which would serve as the content 
of the various evaluation approaches, the project conmiittee was asked to 
generate a total of 14 patient protocols. In their selection of case materials, 
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they were asked to focus upon patients whom if might reasonably be expected 
would be seen by primary care physician's assistants. For this purpose, 
committee members were asked to rely upon case materials drawn from actual 
patient files. In order to guide their development of these patient protocols, 
each project committee member was given a patient protocol outline which 
described the categories of descriptive data they should develop for each 
clinical case. Each resulting patient protocol consisted of a sufficiently 
comprehensive yet succinct description of the patient's current problems and 
life situation, affective profile, medical and psychosocial histories, clinical 
setting, and behaviors that physician's assistants should and should not engage 
in. A sample of the patient protocol outline is supplied in Appendix C. 

All patient protocols were reviewed by the entire committee and modified 
as necessary in order to develop a final protocol which would be a realistic 
yet challenging clinical case for the physician's assistant. 

Simulation Development 

The authors of each patient protocol were then asked to write up to 
seven problem-centered multiple-choice questions per protocol. These test 
questions were reviewed by the project staff and subsequently by the entire 
committee in order to maximize the face validity of each multiple- 
choice question. A total of 82 such questions were finally selected for 
inclusion in the series of multiple-choice questions used for this study. 

The project committee was then asked to select protocols which they felt 
were rich in content and opportunities for the evaluation of skills in patient 
interviewing and counseling. Seven such protocols were identified, and each 
member of the committee was assigned the responsibility of developing a 
branching paper-pencil simulation script based upon one protocol. The resulting 
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seven scripts were reviewed in detail and modified by the committee at a series 
of subsequent meetings. In developing the options to be presented at each 
decision point in the script, an effort was made to present questions and 
comments which reflected different degrees of appropriateness. In addition, 
the content of all options was designed so as to permit as wide a variation in 
interviewing approach as possible within the constraints of the paper-pencil 
methodology. Of the seven scripts reviewed by the committee, five were selected 
for further development and use with the paper-pencil simulations and interactive 
audiovisual simulations. The following is a selected sample of the clinical 
case materials contained in the various patient protocols: 

a. unwed pregnant teenage girl facing conflict with her family regarding 
what she should do about her pregnancy as well as her own uncertainty 
about the course of action she should take; 

b. a male patient who is recovering from a myocardial infarction and 
having difficulties coping with his altered self-image and the need 
to modify his life style; 

c. a married woman with a prolonged upper respiratory infection, secondary 
to marital problems; 

d. an obese male who is on a weight control program and who is having 
difficulty coping with a new self-image and the demands placed upon 
him by his altered peer relationships: 

e. a woman who is denying the fact that her first child has been bom 
with Down* s Syndrome and who has come to the practice seeking 
**another medical opinion'*; 

f . a pregnant woman who is concerned that her mild hypertension in the 
last trimester of pregnancy may result in an unsuccessful outcome 
as has been the case in her first two pregnancies; 
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g, an elderly man who has a series of physical and psychological problems 
secondary to the effects of the aging process. 

Scoring Strategy 

The twenty-eight categories of behavior developed for the interaction 
observation form were used not only to classify behavior observed during the 
live interviews but also to classify options selected on the multiple-choice 
questions > paper-pencil simulations > and interactive audiovisual simulations. 
This procedure was followed so that comparisons could be made among the 
evaluation methodologies. 

These categories of behavior were reviewed and related behaviors were 
clustered together. This clustering resulted in the identification of the 
five scoring variables. 

a. efficiency 

b . non-directiveness 

c. facilitation of expression/rapport - support 

d. enhancing patient understanding 

e. responsiveness to patient behaviors 

Each scoring variable consisted of a positive and a negative dimension. 
Behaviors within each scoring variable were assigned to one or the other 
dimension as a function of whether or not it was felt to contribute to the 
communication goal used to describe the particular scoring variable. Table 4 
illustrates the positive and negative exemplars of behavior associated with each 
scoring variable. 
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Table ^ 

Scoring Variables and Behaviors 
Included in Each 



Variable 1: EFFICIENCY 



Positive 



Asks case pertinent, open-ended question 

Asks case pertinent, close-ended question 

Follows up pertinent verbal or non-verbal 
cues 



Negative 



Asks non-case pertinent, open- 
ended question 

Asks non-case pertinent, close- 
ended question 

Asks repetitious question 

Fails to follow-up pertinent 
verbal or non-verbal cues 



Variable 2: NON-DIRECTIVENESS 



Positive 

Asks case pertinent, open-ended question 

Reflects patient coimranent or question as 
a probe 

Encourages patient to continue talking 



Negative 



Asks case pertinent, close-ended 
question 

Asks case pertinent inappropriately 
leading question 

Asks continuous questions 

Inappropriately interrupts 

patient /abruptly shifts focus 
of interview 

Repeats own question verbatim 
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Variable 3: FACILITATION OF EXPRESSION/ 
RAPPORT - SUPPORT 



Positive 

Reflects patient comment or question as 
a probe 

Encourages patient to continue talking 

Appropriately summarizes /synthesizes 
patient comments 

Expresses understanding 

Reassures patient appropriately 

Expresses empathy 



Negative 

Inappropriately interrupts patient/ 
abruptly shifts focus of interview 

Uses medical jargon 

Inappropriately suiranarizes /synthesizes 
patient comments 

Responds with Inappropriate affect 

Reassures patient Inappropriately 



Variable 4: ENHANCING PATIENT UNDERSTANDING 



Positive 

Clarifies ovm prior question/ comment 
Gives advice/instructions appropriately 
Answers patient questions directly 
Provides appropriate information 



Negative 

Uses medical jargon 

Repeats own question verbatim 

Gives advice/instructions 
inappropriately 

Answers patient questions evasively 

Provides inappropriate information 



Variable 5: RESPONSIVENESS TO PATIENT BEHAVIORS 



Positive 

Reflects patient comment or question as 
a probe 

Encourages patient to continue talking 

Follows up pertinent verbal or non-verbal 
cues 

Expresses understanding 
Reassures patient appropriately 
Answers patient questions directly 
Expresses empathy 

38 



Negative 

Inappropriately interrupts 

patient/abruptly shifts focus 
of interview 

Fails to follow-up pertinent verbal 
or non-verbal cues 

Repeats own question verbatim 

Responds with inappropriate affect 

Reassures patient inappropriately 

Answers patient questions evasively 

Provides inappropriate information 
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The specific behaviors included in each of the scoring variables was 
determined by the combination which yielded the highest intra-class correlation 
coefficient for a given variable. Thus, before determining final composition of 
each scoring variable, a number of intra-clasa correlation coefficients was 
calculated, eliminating behavioral categories that resulted in considerable 
variation in observations among the raters. 

After the number of scoring variables and their composition had been 
determined, attention turned to the development of a scoring formula which 
could provide numerical values reflective of the quality of the behaviors 
associated with each scoring variable. The scoring formula was baaed upon a 
number of principles which were formulated in advance. For example, it was 
felt that the score for each variable should reflect the relationship between 
the number of positive and the number of negative behaviors observed. Moreover, 
ic was felt that the number of negative behaviors should be weighted more heavily 
than the positive behaviors since their Impact upon the interactive process 
is probably greater per instance of behavior than is the impact of each 
positive behavior. With these principles in mind, a number of hypothetical data 
sets were scored using a preliminary scoring formula. This preliminary formula 
was modified somewhat to better reflect the principles stated above, and the 
following represents the scoring formula finally selected: 

^pos, 4- 1 

X » 1,5 (Cneg,) + 2 

where, 

^pos, » sum of the positive 
behaviors 

jTneg, sum of the negative 
behaviors 
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Utilizing the above formula, the following scores would be obtained from the 
data set presented below. 



, POS. 


<,neg. 


X 


0 


10 


.059 


0 


5 


,105 


1 


10 


,118 


0 


1 


.286 


1 


1 


.571 


5 


5 


.631 


10 


10 


.6A7 


1 


0 


1.0 


5 


0 


3.0 


10 


1 


3.1A 


10 


0 


5.5 



All items cortained in the multiple-choice questions, paper-pencil 
simulations, and audiovisual simulations were reviewed by a member of the 
project staff and classified into one or more behavioral categories. Thus, 
the profile of behaviors produced by each subject could be compared across 
all evaluation methodologies. The scoring formula presented above was used 
to calculate scores for each variable on all evaluation methodologies. 

Reliabi lity Studies - Interaction Observation Form 

Before using behavior in a live interactive setting as the criteiion 
measure in this study, two inter-rater reliability studies of the interaction 
observation form were conducted. It was felt that the best indicator of the 
stability of the measurements derived from this fo.m would be the intra-class 
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correlation coefficient. This statistic is an estimate of the average agreement 
among multiple observers providing data on the same interviewing or counseling 
session. 

In order to achieve a maximum standardization of the content of these live 
interviews in the first study, four patient protocols were selected and individuals 
were recruited to portray each role. The individuals selected were non-professional 
actors from conmiunity acting groups and the University of Pennsylvania School 
of the Performing Arts, Each actor and actress was given a period of orientation 
and training in portrayal of their respective roles. 

Eight physician's assistants from a local training program volunteered to 
participate in the reliability study by interviewing the programmed patients. 
Each physician*s assistant interviewed two of the four programmed patients, and 
each interview was videotaped using two hidden cameras, A total of sixteen 
videotaped interviews was available for analysis. 

The project conmiittee was brought to Philadelphia where they were given a 
half-day training program in the use of the interaction observation form. They 
were also asked to critique the forms and offer suggestions for modification. 
These suggestions were incorporated into the forms, and the committee was then 
asked to view four videotaped interviews and to individually classify the 
behaviois on the modified interaction observation form. Committee members 
viewed each videotape simultaneously but individually completed the form. Thus 
for each of the videotaped -interviews there were six interaction observation 
forms , 

The twenty-eight behavioral categories on the observation form had been 
clustered into five scoring variables ^cr purposes of evaluating physician's 
assistant behavior. These scoring variables were labeled as follows: 
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a. efficiency 

b . non-directiveness 

c. facilitation of expression/rapport - support 

d. enhancing patient understanding 

e. responsiveness to patient behavior. 

An intra-class correlation coefficient was calculated for each of the 
five variables. For this first reliability study, these correlation coefficients 
ranged from .70 tu .87. It was felt, however, that with further training of 
the raters these reliability coefficients could be increased and provision 
was made to conduct a second reliability study. 

The second reliability study followed the same format as the first, but 
a more systematic approach was used in training the committee members to use 
the behavioral categories. A different set of four videotaped interviews was 
shown. 

An intra-class correlation coefficient was calculated for each scoring 
variable and, as had been hoped, four of the five reliability estimates increased. 
In the one instance where there was a slight decrease in the reliability estimate, 
it was not felt to be a significant difference. The reliability estimates for 
the first and second reliability studies are presented in Table 5 . In the 
first study, the intra-class correlation coefficient was calculated using six 
observers, while in the second study five observers were present. 

The results of the second reliability study suggested that the interaction 
observation form did yield sufficiently reliable estimates of performance to 
warrant its use as a criterion measure in the overall research design. 

\ 
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Table 5 



Reljfibility Estimates^ of 
Interaction Observation Fonn 



Variable 



Study //l^ 



Study //2^ 



Efficiency 

Non-Di rec tiveness 

Facilitation of Expression/ 
Rapport - Support 

Enhancing Patient 
Understanding 

Responsiveness to Patient 
Behavio rs 



.72 

.70 
.77 

.87 

.70 



.86 
.93 
. 82 

.82 

.84 



%ased upon intra-class correlation coefficient 
^using 6 jude.es 
^using 5 judges 

Resear ch Design and Data Collection 

Subjects for this study were drawn from three physician's assistant 
training programs. All subjects were volunteers who were paid a modest 
honorarium for theii participation in the study. A total of 59 subjects 
participated, nineteen at site one and twenty each from sites two and three. 
All subjects were evaluated using all evaluation approaches* 

In order to control for the sequence in which subjects would experience 
the various methodologies, the subjects at each site were randomly assigned 
to one of four groups. The sequence in which each group experienced the 
evaluation methodologies is presented below; 
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Group A : 



Paper-pencil simulations; multiple-choice questions; audio- 
visual simulations; live interviews 



Group B: 



Multiple-choice questions; paper-pencil simulations; live 
interviews; audiovisual simulations 



Group C : 



Audiovisual simulations; live interviews; multiple-choice 
questions; paper-pencil simulations 



Group D : 



Live interviews, audiovisual simulations; paper-pencil simulations; 
multiple- choice questions 



For the live interviews, subjects within each group interviewed trie programmed 
patients in a randomly determined sequence. 

The content for the live interviews was selected from five patient 
protocols.. Non-professional actors were selected and trained to portray a 
specified role. The period of time required for orientation and training of 
each programmed patient equaled approximately four hours. 

The studies were conducted at each site during a two-day period. Each 
subject experienced two evaluation methodologies the first day and the 
remaining two methodologies during the second day. Each live interview was 
audio recorded for subsequent analysis by members of the project committee. 
Subjects were told they were participating in an experimental study out were 
not informed that the purpose of the study was to assess interpersonal and 
communication skills.- 

Following the completion of the study, a different set of audio tapes was 
sent to each member of the project committee along with retraining materials, 
hlank interaction observation forms, and instructions for analyzing the tapes. 
Members of the committee did not analyze audio recordings which were made at 
their own physician's assistant training program. A total of 295 audio 
recordings were analyzed. Since members of the project committee had 
participated in the two reliability studies, it was felt that this experience 
along with the retraining materials provided an adequate background for analysis 
of the audio recordings. A A 



Flndings 

Table 6 displays the correlations among the five interpersonal skills 
variables as assessed by the live interviews. Efficiency does not correlate 
significantly with any of the other variables except Responsiveness to Patient 
Behaviors (Variable //5 , .365). In fact, all interpersonal skills variables 
correlate more highly with Responsiveness to Patient Behaviors than with any 
other variable. 

The explanation of the high correlations among Variables it2 through 115 lies 
in the distribution of behaviors across variables. For example, of the 
8 behaviors included in Variable ill (Non-Directiveness) , 3 are also included 
in Variable //3 (Facilitation of Expression/Rapport-Support) and 4 are included 
in Variable //5 (Responsiveness to Patient Behaviors). One would hypothesize, 
then, that the correlation between Variables //2 and //5 would be greater than 
that between Variables //2 and //4. These hypochesized relationships are, 
in fact, obsc^p<^ in Table 6. The coefficient between Variables //2 and //5 
equals . 732, while the correlation between Variables 1}2 and //A equals .459. 

A similar pattern of relationships can be observed between Variables //3 
(Facilitation of Expression/Rapport -Support and Variable 115 (Responsiveness), 
the 11 behaviors included in Variable //3, eight are also included in Variable //5. 
Thus, it is not surprising to find that the correlation between these two 
variables equals .905. 

The distribution of interpersonal skills behaviors across the five scoring 
variables is displayed in Table 7. In interpreting the magnitude of relationships 
among the five interpersonal skills variables, it should be remembered that some 
are particularly inflated due to the overlap in content across variables. The 
correlations to which this caveat applies have been underlined in Table 6. 



Table 6 



Correlations Among Interpersonal Skills Variables 



LIVE INTERVIEWS 



1 2 3 4 5 



2 .162 

3 .191 **.682 

4 .191 **.459 **.727 

5^ **.356 * *.732 ** .905 * *. 778 

**p^.01 

df = 57 
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Table 7 



Distribution of Interpersonal Skills 
Behaviors Across Scoring Variables 



1, Non-pertinent, open-ended questions 

2. Pertinent, open-ended questions 

3» Non-pertinent, closed-ended questions 

4, Pertinent, closed-ended questions 

5. Inappropriately leading question 
6v Reflecting as a probe 

7. Repetitious questions 

8. Continuous questions 

9. Interrupts - abruptly shifts focus 

10, Encourages patient to talk 

11, Uses medical jargon 

12, Follows-up cues 

13, Fails to follow-up cues 

14, Repeats question verbatim 

15, Clarifies question 

16, Appropriately summarizes 

17, Inappropriately summarizes 

18, Expresses understanding 

19, Responds with inappropriate affect 

20, Reassures appropriately 

21, Reassures inappropriately 

22, Answers questions directly 

23, Answers questions evasively 

24, Gives appropriate advice/instructions 

25, Gives inappropriate advice/instructions 

26, Provides appropriate information 

27, Provides inappropriate information 

28, Expresses empathy 



X 
X 
X 
X 



Variable 
2 3 A 



X 
X 
X 

X 

X 
X 



X 
X 



X 
X 
X 



X 
X 
X 
X 
X 
X 



X 
X 



X 
X 
X 
X 
X 
X 



X 
X 

X 
X 
X 



X 
X 
X 
X 
X 
X 



X 
X 
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It is interesting to note that Efficiency does not correlate significantly 
with Non-Directiveness , Facilitation of Expression/Rapport-Support, or 
Enhancing Patient Understanding. There is a modest correlation between 
Efficiency and Responsiveness to Patient Behaviors (.356, p ^.01), but the 
correlation may be due solely to the fact that 2 of the 6 behaviors contained 
in the Efficiency variable (following and failing to followup patient cues) 
are also included in the Responsiveness variable. Thus, it appears that 
efficiency does not account for a significant proportion of the variance 
included in the other more affective-laden interpersonal skills variables. 

Although the major thrust of this study was not the investigation of 
relationships among the 28 interpersonal skills behaviors, inter-correlations 
among the 28 variables as measured in live interviews, were calculated in 
order to determine the extent of their independence. Table 8 contains a 
summary of the best predictors for each of the 28 interpersonal skills behaviors 
as depicted from the live interviews. All coefficients are at or beyond the 
,05 level of confidence. As can be seen from this table, even when an effort 
is made to focus on best predictors, the relationships among these variables 
are quite modest. Only Variables //2 , //9 , //16 and #18 show correlation 
coefficients above .50, Thus, it seems reasonable to conclude that interpersonal 
skills behaviors are only modestly correlated among one another. There are, 
however, some Interesting relationships among behaviors which, while not 
surprisingly large in terms of magnitude, are logically consistent. Some 
examples of such relationships are as follows: 

a. The more interviewers express empathy to patients, the more they 
express understanding of what the patient has said and reassure 
the patient appropriately. Conversely, the more interviewers 
express empathy, the less they inappropriately interrupt patients 
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or abruptly shift focus and answer patient questions evasively. 

b. There is a positive relationship between providing inappropriate 
information and answering patient questions evasively, 

c. There is a positive relationship between providing appropriate 
information and the extent to which the interviewer reflects patient 
connnents or questions as a probe and appropriately summarizes 

or synthesizes patient comments. There is a negative relationship 
between providing appropriate information and responding to patient 
j comments with inappropriate affect, 

d. There is a positive relationship between expressing understanding 
of what the patient has said and appropriately summarizing and 
synthesizing patient comments. There is a negative relationship, 
however, with expressing understanding and inappropriately interrupting 

patients or abruptly shifting focus, 

I 

e. The more interviewers use medical jargon, the less willing they are 
to clarify their own questions and comments to the patient, 

f. The more interviewers encourage patients to continue talking, the 
more they answer questions directly and the less they respond to 
patients with inappropriate affect. 

g. The more interviewers interrupt patients and abruptly shift focus, 
the less likely they are to appropriately summarize and synthesize 
patient comments and express understanding of what the patient has 
said, 

h. The more repetitious questions the interviewer asks, the less likely 
he is to give appropriate advice and instructions, 

i. The more interviewers use inappropriately leading questions, the 
more likely they are to inappropriately interrupt patients and 
express inappropriate affect in response to patient comments. 
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Table 8 . ^ i^'^] 

Correlations Among 28 Interpersonal Skills Behaviors Depicted in Live Inter /lews (p^05) 

3 4 5 G 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 

-.36 

.27 

-.33 

-.29 

-.26 .41 .35 .38 



-.34 -.30 



9 .40 -.33 



10 .23 

1 I 



-.61 -.26 -.54 .38 .34 -.43 -.38 

«.37 .39 
.35 -.44 -33 

-.34 



17 
iS 



-.44 

-.61 
-.51 



.54 



.36 



1 V .11 -.37 

l-vO - . 30 - . 33 
J I 

22 .39 -.34 

21 -,33 -.43 -.33 

25 

21 -.30 



54 .39 -.35 -.40 .36 

-.35 

.35 

-.36 

-.36 

,30 -.36 

-.36 

-.36 -.31 
-.31 

.35 -.36 
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Since the major thrust of this project was to evaluate the effectiveness 
of several indirect testing techniques in predicting performance as assessed 
in a live interview setting, within and between-met hod correlation coefficients 
were calculated across the 5 interpersonal skills variables. These correlations 
are presented in Table 9. From the standpoint of within-me thod validity, the 
best predictor of each live interview variable should be the corresponding 
variable on the indirect technique. The indirect technique which produces 
the largest number of such corresponding correlations can be thought of as 
the most valid. 

For the interactive audiovisual simulations, the live interview Efficiency 
score was best predicted by the corresponding audiovisual simulation Efficiency 
variable. For the live interview variable, Enhancing Patient Understanding, 
the corresponding audiovisual simulation variable also yielded the best prediction. 
The same was found to be true in relation to the variable. Responsiveness to 
Patient Behaviors, on the live interview and the corresponding audiovisual 
simulation variable. Thus, for the interactive audiovisual simulations, the 
appropriate pattern of validity coefficients was present in 3 of the 5 interpersonal 
skills variables* 

For the paper-pencil simulations, there was a correspondence between 
live interview variables and the corresponding variables on paper-pencil 
simulations in only i out of 5 instances: Facilitation of Expression/Rapport- 
Support , 

For the multiple-choice questions, the correspondence between scoring 
variables was observed in only 1 instance, and involved the Responsiveness to 
Patient Behaviors variable. 

Thus, in relation to validity coefficients, interactive audiovisual 
simulations yielded the largest number (3) as compared with paper-pencil 
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efficiency 
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1. Efficiency 
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.209 
.475 
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.329 



.309 
.•39 



.713 
.399 



MP 
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.243 

.131 

.129 



.429 

.370 
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.333 
.406 
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.303 
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.373 
.309 .094 
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,301 
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.139 
.299 

.323 

.293 
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.077 

.044 

.214 

.092 



.119 
.234 
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.197 .224 
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.233 
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,229 
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.141 


.191 
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.230 
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.191 
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.203 
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.335 -.199 
.S5S -.133 
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.914 
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,300 
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.299 
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*The unlta of analyala vera coapllatlona of raaka, tranaform^d to atandard »cor»», of aubMta of th» ^9 behavioral oat»worl»a. 
Ne« York: Holt. Rln«hart 4 WlnttoQ. 1933. p. 424). 
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simulations and multiple-choice questions which yielded 1 each. The 
coefficients which provide between-method indications of validity have 
been circled on Table 9. 

Table 10 shows the simple correlations between the 5 live interview 
variables and the corresponding variables for each indirect testing technique. 
As can be seen from this table, none of the Efficiency scores on the indirect 
techniques correlate significantly with the Efficiency score in live interview 
settings. The correlation coefficient of .21 between Efficiency in the live 
interview setting and Efficiency as measured by the interactive audiovisual 
simulations comes closest to reaching statl,stical significance at the .05 level 
of confidence (r of .25). In relation to Non~Directiveness , the corresponding 
variable as measured by multiple-choice questions yielded the largest correlation 
coefficient (.325). This coefficient is significant beyond the .01 level of 
confidence. In terms of Faciliation of Expression/Rapport-Support, Enhancing 
Patient Understanding, and Responsiveness to Patient Behaviors, the corresponding 
variables as assessed by interactive audiovisual simulations yielded the largest 
correlation coefficients. These coefficients equaled .332, .372, and .A26 
respectively and are all significant beyond the .01 level of confidence. 

In summary. Table 9 indicates that 4 of the 5 interpersonal skills 
variables can be best predicted by performance as assessed by interactive 
audiovisual simulations. In one instance, (Non-Direct iveness) this prediction 
was best accomplished through the use of multiple-choice questions. It should 
be again pointed out that the prediction of each interpersonal skills variable 
was limited to the corresponding variable as measured by each of the indirect 
testing techniques. 
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Table 10 



Simple CorreJations Between 
Live Interview Variables 
and Testing Techniques 



Multiple-Choice Questions 



,013 



**.325 



.224 



.1A5 



*. 317 



Paper-pencil Simulations 



-.049 



*.253 



*.289 



.018 



*. 261 



I 
I 



Audiovisual Simulations 



210 



.086 



**.332 



**.372 



**. A26 



*p = .05 
**p «i .01 

df = 57 
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In order to estimate which variables across testing techniques would 
be the best predictors of each live interview variable, a step-wise multiple 
regression analysis was performed. The results are presented in Table 11. 
There was no scoring variable or indirect testing technique which would predict 
Efficiency at the .05 level of confidence. For this reason, it was not 
possible to obtain a regression solution for this variable. The other results 
of the regression analysis indicate that Non-Directiveness can be best 
predicted by combining the Responsiveness scores as measured by multiple- 
choice questions and interactive audiovisual simulations. Under these 
circunistances, the multiple correlation coefficient equal.^5 .485. Similarly, 
it was found that the best predictor of l*'acil J tation of Expression/Rapport- 
Support is also the Responsiveness score as assessed by interactive 
audiovisual simulations, resulting in a correlation coefficient which 
equals .450. The best predictor of Enhancing Patient Understanding is the 
corresponding variabJe as assessed by interactive audiovisual simulations 
plus Facilitation of Expression/Rapport-Support also as measured by 
interactive audiovisual simulations. When these two predictors are combined, 
they result in a multiple correlation coefficient equal to .462. . 
Finally, Responsiveness to Patient Behaviors is best predicted by the 
corresponding variable as assessed by interactive audiovisual simulations. 
This correlation coefficient equals .425. 

Perhaps the most striking aspect of the multiple regression analysis 
is the fact that of the 6 predictors identified, 5 are obtained from 
assessment by the Interactive Audiovisual Simulation System. Moreover, of 
the 5 scoring variables into which behaviors were classified. Responsiveness 
to Patient Behaviors appears to be the most potent predictor of other 
variables as well. An explanation for this phenomenon can be found in the 
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Table 11 

Best Predictors of Variables in Live Interview 



Variable in Live 
Interview** 

2 . Non-Directiveness 



Predictors 

MCQ Responsiveness 
AVS Responsiveness 



Multiple 
Correlation* 

.AOO 
.A85 



Simple 
Correlation* 

,A00 
,398 



Facilitation of 
Exp res si on /Rapport 
Support 



AVS Responsiveness 



450 



.450 



Enhancing Patient 
Understanding 



AVS Enhancing Understanding 
AVS Facilitation/Rapport 



371 

,462 



,371 
.332 



Responsiveness to 
Patient Behaviors 



AVS Responsiveness 



,425 



.425 



*all coefficients are singificant beyond .01 level of confidence 
**F-level was not sufficient for the Efficiency variabJe to permit a solution 
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fact that the Responsiveness variable contains 13 of the 28 categories of 
ir terpersonal skills behaviors. 

The fact that Facilitation of Exprassion/Rappor t-Support is best 
predicted by Responsiveness to Patient Behaviors (measured by interactive 
audiovisual simulations) , may be explained by the large amount of overlap 
in content between these two variables and the possibility that there is 
significant error variance in the behaviors which are included in the 
Facilitation variable but not in the Responsiveness variable, 

A similar phenomenon may also be present with respect to the Non-Direct ive- 
ness variable in that the non-overlapping behaviors may have little variance 
or considerable error variance associated with them. If such were the case. 
Responsiveness to Patient Behaviors could be expected to be the best predictor 
of Non-Direct iveness , 

Conclusions 

The data presented above suggests that: 

a. There is greater evidence for the convergent and divergent validity 
of interactive audiovisual simulations than for multiple-choice 
questions or paper-pencil simulations, 

b. In predicting interpersonal skills behaviors as measured in a live 

interview setting, the corresponding variables are best predicted 

in A out of 5 instances by interactive audiovisual simulations when 

the basis for prediction is a simple correlation coefficient'. In 

predicting the remaining variable (Non-Direct iveness) with o 

corresponding variable from ai indirect testing technique, the use 

of multiple-choice questions yields the largest correlation 

coefficient. ^ 
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c. In predicting live interview variables across testing techniques 
by means of multiple regression, interactive audiovisual simulation 
is a significant predictor of all variables which were included in 
the analysis. In addition to interactive audiovisual simulation, 
Responsiveness as measured by multiple-choice questions emerged as 
the only significant predictor variable from the other two indirect 
testing techniques investigated in this study. 
It seems reasonable to conclude that of the three indirect testing techniqu 
investigated in this study, interactive audiovisual simulations yield the 
most potent predictions of performance in a live interactive setting. The 
magnitude of these correlations (,40 to ,45) suggests that additional work 
should be undertaken in order to further refine this innovative technique 
and to determine the maximum level of prediction possible. In view of the 
fact that the interactive audiovisual simulations and the corresponding 
Interactive Audiovisual Simulation System are in prototype stages of 
conceptualization and -development, the magnitude of thp correlation coefficients 
is particularly encouraging.^ It suggests that further refinement of this 
evaluation methodology could significantly enhance its contribution as an 
approach to the assessment of the interpersonal skills of health prof essionals ,> 
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AUDIT VALIDATION STUDY 

Introduct ion 

National examinations for licensure and certification in the health 
professions have traditionally been used as one of the components in evaluating 
qualifications for practice. The major purpose of these examinations is to 
identify candidates who are not yet ready to assume responsibility for practice 
in a given health profession. In essence, such examinations serve as a fail- 
safe mechanism for protecting public safet.y and the integrity of the profession 
itself. Since it is generally assumed that the quality of a profession is only 
as good as the skills of the individuals who practice it, the maintenance of high 
standards of practice is essential. 

Because of the role that licensing and certifying examinations play in this 
process, their structure and content need to be continually evaluated in order to 
ensure that they are measuring knowledge and skills relevant to practice in tha 
profession.^ It is assumed that if a certifying examination does, indeed, assess 
relevant aspects of competence, then performance on the examination should have 
a positive relationship to subsequent performance in practice. 

It is doubtful that anyone would argue the merits of the above assumption. 
What does become a point of some debate is the nature of the specific relationships 
that should be expected, and the magnitude of these relationships. If one returns, 
for a moment, to the role of certifying examinations, one finds that they are 
used only as one component in an overall evaluation process. The reason that 
such examinations are not used as the sole component is twofold. First, a 
single evaluation program cannot be expected to measure all of the areas of 
knowledge and skill required for competent practice. Secondly, no measure of 
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competence, be it grades, formal examinations, faculty ratings of performance or 
actual observations of performance yields perfect predictions of subsequent 
behavior in a practice setting. Thus, using several sources of data in evaluating 
qualifications for practice serves to enhance the accuracy of the decision to 
grant or withhold certification. By ensuring that there is a positive relationship 
between performance on these input measures and subsequent performance in practice, 
we help ensure the validity of our evaluation procedures. Certifying examinations 
will not be able to assess all relevant components of competence, and for 
those components they do assess, they will not do so perfectly. 

These observations are in no way meant to serve as an apologia for certifying 
examinations. Rather they are meant to place these examinations into perspective 
so that reasonable expectations can be specified and tested. 

The Physician's Assistant Certifying Examination has undergone a number of 
validation studies during its first three years of implementation. These 
studies have shown significant positive relationships between scores on various 
components of the Certifying Examination and faculty ratings of these components 
of competence. While the magnitude of these relationships has not exceeded ,A0, 
the patterns of significant correlation coefficients indicate stronger relationships 
where one would expect to find them and only chance level relationships where 
logically one would not expect to find a correlation between examination scores 
and ratings of clinical competence. In addition, physician's assistants with up 
to two years of clinical experience following completion of an educational 
program achieve significantly higher examination scores than candidates who sit 
for the examination before completing their training program. This finding 
suggests that the examination is measuring knowledge and skills that are relevant 
to practice since candidates with practice experience achieve significantly 
higher scores on the examination. 
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An important relationship that has not yet been explored, however, is the 
extent to which performance on the Certifying Examination is predictive of certain 
aspects of day-to-day performance in an actual clinical practice. In order to 
investigate this relationship, the National Board of Medical Examiners, with the 
endorsement of the American Academy of Physicians' Assistants, embarked upon a 
study to obtain measures ot pertormancf* in an actual practice setting so that 
these could be correlated with scores on the Certifying Examination. 

Obviously, the assessment of performance in a practice setting is a vgry 
complex endeavor. The assessment procedure eniployed must not only yield valid 
and reliable observations of performance, but it must focus on relevant aspects 
of practice behavior in a manner that is unobtrusive and cost effective. 

How does one determine whether an adequate history and physical examiration 
were performed and the appropriate laboratory and diagnostic studies ordered? 
How does one evaluate the physician assistant's skill in interpreting these 
clinical findings? How doe5 one determine whether the P.A. uses appropriate 
judgment in seeking consultations and referral? How can one assess whether 
appropriate management and treatment was instituted and adequate instructions 
provided to patients? 

If these are the major components of competence we wish to use as criteria 
for evaluating practice performance, we need to consider the evaluation 
methodologies that are available for the collection of these criterion data. 
Three methodologies exist: a) direct observation of P,A. performance by a 
trained observer; b) employer ratings of P.A, performance; and c) reviews of 
office medical records. 

Each of these methodologies poses certain limitations and problems for the 
evaluation of practice performance. For example, direct observation of 
performance involves significant amounts of time and expense. Moreover, the 
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presence of an observer requires the patient's consent and can bias the P.A.'s 
behavior. Employer ratings of performance can be more time and cost effective 
than the previous methodology cited, but are subject to the same problems 
generally associated with the use of rating scales — such as, a tendr^ncy to 
be very lenient in the ratings of performance, and to rate an individual In 
the same way across the various competency components under consideration. 
Because of the subjectivity of these ratings, they may not provide accurate 
evaluations of clinical performance. Finally, the review of office medical 
records provides a reasonably objective basis for documenting behavior, 
although it does not provide a mechanism for evaluating the physician assistant's 
skill in interpreting clinical data or judgment in seeking advice and consultation. 
Moreover, the medical record may not fully document all pertinent aspects of 
the evaluation and management of a given patient. Thus, in order to use the 
medical record as a basis for documenting performance, one must assume that the 
record provides a reasonably accurate documentation of clinical behavior. 

VAien cost effectiveness and objectivity of Che evaluation process were 
considered for each of the methodologies noted above, it appeared that the review 
of office medical records could provide the best source of data regarding 
practice performance. 

The review of office medical records is based upon the identification of 
specific medical conditions which serve as the guide for selecting appropriate 
charts for review. The conditions selected should include those which are seen in 
the practice setting with sufficient frequency to provide an adequate sample of 
medical records. Since the Certifying Examination focuses on primary care> it 
was felt that the evaluation of practice performance should be based upon 
physician* s assistants who are practicing in primary care settings. 
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Proced ures 

Having selected medical record audits as the methodology to be used in gathering 
data regarding practice performance, several objectives needed to be accomplished. 
The m^edical conditions and associated audit criteria had to be identified; these 
criteria needed to be pilot-tested in order to ensure their appropriateness 
and clarity; individuals who would be responsible for auditing the office medical 
records had to be identified and trained; and procedures needed to be 
established to identify appropriate medical records within each practice and 
to abstract the data contained in each. 

In order to maximize participation in this study, an endorsement was 
obtained from the American Academy of Physicians' Assistants, In addition, 
the Executive Committee of the Academy was asked to nominate physicians and 
physician's assistants who could serve on the Advisory Committee. The role of 
the Advisory Committee was to identify the specific medical conditions to be 
used in this study and to specify the audit criteria for each, 

A ten-member Advisory Committee was appointed consisting of practicing 
physician's assistants, primary care physicians who employed physician's 
assistants, and physicians involved In the training of physician's assistants. 
At its first meeting, the Committee was asked to identify a number of medical 
conditions which they felt were representative of those seen in primary care 
practice settings. The Committee identified the following twelve conditions: 

a, angina - initial and follow-up evaluations 

b, asthma - initial and follow-up evaluations 

c, birth control pills - initial and follow-up evaluations 

d, diabetes mellitus - initial and tollow-up evaluations 

e, headache 

f, health maintenance 

g, hypertension - initial and follow-up evaluations 

(i7 
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h, acute purulent otitis media 

i , to nsillo pharyngitis 

j, urinary tract Infection 

k. vaginitis 

1. venereal disease 

Each member of the Advisory Committee was assigned the responsibility 
of developing audit criteria for several of the medical conditions so that 
these criteria could be reviewed and refined by the entire Committee at subsequent 
meetings. Committee members were advised that the audit criteria should consist 
of procedures felt to be essential to the diagnostic work-up and management of 
patients with the specific medical condition. Essential procedures were felt 
to be those which were necessary to establish the correct diagnosis and to 
resolve the patient's medical problem. 

For each medical condition, historical, physical examination, laboratory 
data, and management plans that should be found in the medical records of 
patients being seen for evaluation were specified, (A sample of the audit 
criteria specified for two of the medical conditions is provided in Appendix D,) 
In addition to the criteria themselves, examples of acceptable and unacceptable 
chart entries were also specified so that record abstractors could determine 
whether a specific chart entry satisfied a given criterion. Both the audit 
criteria and the examples of chart entries were specified on the basis of 
consensus reached by the Advisory Committee after careful discussion and pilot 
use of the criteria with a sample of medical records. (The pilot use of 
the criteria was carried out by the project staff, using anonymous medical 
records thnt had been provided by individual members of the Advisory Committee.) 
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An abstracting form was prepared for each set of audit criteria. The form 
was designed so that record abstractors could record whether or not a given 
medical record satisifed each criterion specified for a given medical condition. 
(Several sample abstracting forms and instructions for abstractors are provided 
in Appendix E.) For each criterion the abstractor was required to make one of 
four decisions: "Yes,'* if the chart entry satisfied the criterion; "No,'* if 
the chart entry did not satisfy the^ criterion; "UC," if the abstractor was 
uncertain as to whether or not the chart entry met the criterion; "NA," if 
the criterion was not applicable for the specific patient whose medical record 
was being audited. (Such instances were often related to age and sex variables, 
or the presence or absence of specific diagnosis findings.) 

In order to obtain an adequate sample of medical records for each 
participant in the study as well as to facilitate the retrieval of these 
records, a practice log form was developed and sent to each physician's 
assistant. Each participant was instructed to use the log for two months 
during which lime he/she would record the names or Identification numbers of 
patients who were seen for one or more of the medical conditions included in 
this study. When the spaces in the log form were filled, the physician's 
assistant was asked to tally the number of patients who had been seen for 
each medical condition and forward this suirmary to the project staff. The 
log form itself was retained in the practice for use as a guide in subsequent 
record retrieval. 

The privacy of patients whose medical records would be audited was 
protected by withholding from the project staff any information concerning 
patient identity. In order to ensure that this procedure was followed, each 
participating physician's assistant was asked to review the abstracting 
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forms before they were forwarded to the National Board to determine that no 
patient identifications had been recorded. In addition, all abstractors were 
cautioned about the need to maintain strict anonymity of all patient data. 

Abstractor Training and Reliability Study 

Seven record abstractors were recruited from around the country and 
brought to Philadelphia for a one-week training program. All abstractors 
were registex"ed record room technicians or administrators who had previous 
experience in auditing medical records. 

During the one-week training program, the audit criteria and examples of 
chart entries for each medical condition were presented and reviewed In detail 
by a physician (and in one instance, a physician's assistant). Abstractors 
were encouraged to ask questions so that ambiguities could be clarified and 
an optimum level of understanding achieved. Following the discussion of each 
set of audit criteria, the abstractors were given several medical records and 
asked to audit each using the appropriate abstracting form. This provided 
abstractors with the opportunity to practice with each set of criteria and to 
identify areas requiring further clarification. This process was repeated 
for each medical condition. 

At the conclusion of the training program, a reliability Scudy was con- 
ducted in order to determine whether the abstractors were functioning with 
sufficient accuracy. Since time would not permit the abstractors to complete a 
set of medical records for all twelve medical conditions included in the study, 
eight medical conditions were selected. During the reliability study, the 
abstractors were instructed not to speak with one another or to discuss 
either the medical records or audit criteria. A member of the project staff 
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remained in the room with the abstractors during the entire study in order to 
ensure that the rules of conduct were followed. 

In order to obtain an estimate of reliability, the coefficient of 
inter-abstractor agreement proposed by Maxwell and Pilliner was used. This 
coefficient is the equivalent c t* an intra-class correlation, but has 
additional advantages in that it is based on individual item responses rather 
than on an overall score. Ideally, a coefficient of abstractor agreement 
should be greater than zero as the amount of agreement departs from chance 
level, and should fall below zero if one abstractor records his judgment regarding 
a criterion in the opposite direction from another abstractor using the same 
criterion. The Maxwell and Pilliner correlation coefficient equals 1.00 only 
if there is complete agreement between two abstractors, and equals 0 if the 
amount of agreement between abstractors could be expected by chance alone. 
The coefficient equals -1.00 if chere is complete disagreement between two 
abstractors using the same criteria in relation to the same medical records. 

Table 12 presents the correlation coefficients between indicated pairs of 
abstractors for eight medical conditions. The overall correlation coefficients 
range from .63 to .79 with a mean of .72. These findings are consistent with 
levels of inter-abstractor agreement observed in similar audit projects at 
the National Board. 

Following completion of the training program, each abstractor was assigned 
a group of practices in a specified region of the country. All participants , 
bad been advised that a project abstractor would be visiting their practice 
within the next several months. Each participant was then advised of the 
name of the abstractor who would visit his/her practice, and each abstractor 
was given a letter of introduction to serve as a means of appropriate 
identification. n i 
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Table 12 

ABSTRACTOR RELIABILITY ESTIMATES 
Audit Validation Study 
Abstractor Pairs Data Base 
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Sample Selection 

A stratified random sample of 800 physician^ s assistants who tool* the 
1974 Certifying Examination was selected for this study. The variables used 
to stratify the sample included: 

a. type of training - whether formally or informally trained; 

b. type of formal training - that is, physician's assistant or Medex 
program;' and 

c. composite examination score 

Accordingly, 800 letters of invitation were sent including a description 
of the purpose of the study and the procedures which would be used to c nduct 
it. In all instances, the American Academy of Physicians' Assistants was 
asked to supply current addresses for those individuals who were members of 
the Academy, Of the 800 physician* s assistants invited to participate in the 
study, 222 indicated that thsy did not wish to do so. Sixty-one physician's 
assistants could not be located by mail or had moved without leaving a forwarding 
address. In these instances, letters of invitation or follow-up letters were 
returned to the National Board and further attempts to obtain a current mailing 
address were unsuccessful. Two hundred eighty-seven physician's assistants 
did not respond to either the initial letter of invitation or the follow-up 
letter. Since these letters were not returned by the post office, it was 
assumed that they had been received. Finally, 230 physician's assistants 
indicated their willingness to participate in the study. Of that number, 96 
dropped from the study for a variety of reasons, leaving a final sample of 
13A physician's assistants whose records were audited. These aspects of the 
sample size are summarized in Table 13. 
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Table 13 

Summary of Sample Size 
Audit Validation Study 



Category 


Number 


Letter of Invitation Sent 


800 


Could not be Located 


61 


Did not Reply 


287 


Unwilling to Participate 


222 


Willing to participate 


230 


Dropped during Study 


96 


FINAL SAMPLE SIZE 


134 



The ninety- six participants who dropped out of the study after indicating 
their willingness to participate did so for a variety of reasons. After being 
advised of the medical coriditions included in this study, twenty-one physician^ s 
assistants indicated that they were not in primary care practice. In eighteen 
instances, physician^ s assistants could not part.icipate further because the 
necessary medical records were not available. Eleven physician *s assistants 
reported that at present they were not in active practice, and ten indicated 
that they were in the process of changing practices. Nine physician*s assistants 
were located in remote sites such as Alaska, Hawaii, and Germany, and it was 
determined that it would not be cost effective for these individuals to 
participate in the study. Five physician^ s assistants subsequently reported 
that they were unemployed and 18 subsequently declined to participate further 
in the study. These reasons for dropping out of the study are summarized in 
Table lA. n ,^ 
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Table 14 
Reasons for Dropping from Study 



Reason Number 

In specialty practice 21 

Records not available 18 

Declined to participate further 17 

Not in active practice 11 

In process of relocating 10 

In a remote site (Alaska, Hawaii, etc.) 9 

Unemployed 5 

Miscellaneous 5 

TOTAL 96 



Of the 134 physician's assistants who participated in all aspects of the 
audit validation study, 87 per cent were graduates of formal educational programs 
and 13 per cent had qualified for the Certifying Examination on the" basis of work 
experience. Of those trained in formal educational programs, 71 per cent were 
trained in physician's assistant programs and 18 per cent in Medex programs. 
(Data were not available for 11 per cent of the participants.) In terms of 
the examination performance of this group of participants, their mean written 
examination score equaled 508 with a standard deviation of 76. The 1974 examina- 
tion statistics for the total examinee population were as follows: the mean 
written scote equaled 492, while the standard deviation equaled 83. The 
characteristics of the study sample in relation to the total examinee population 
are summarized in Table 15. 
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Table 15 



Representativeness of Study Sample 
to Total Examinee Population 



Examinee Study 
Population Sample 



Demographic 


N 


% 


N 


1 


Formally Trained 


1118 


91 




86 


Physician's Assistants 


922 


75 


95 


71 


Med ex 


196 


16 


21 


16 


Informally Trained 


115 


09 


18 


13 


Exam Statistics 


Written 


PAS* 


Written 


PAS* 


Composite Mean 


492 


485 


508 


A 85 


Standard Deyiation 


83 


112 


76 


110 



^Physical Examination skill assessment 



It appears from the data concerning demographic characteristics and 
examination statistics of the study sample that it is reasonably representative 
of the total examinee population for the 197A Physician's Assistant Certifying 
Examination. However, it does represent only 10% of the number of individuals 
who sat as candidates for that examination, and represents 17% of the initial 
stratified random sample of 800 physician* s assistants who were invited to 
participate in the study. The fact that 509 individuals either did not reply 
to our initial and follow-up letters or indicated that they were unwilling to 
participate is likely due in large measure to the sensitive nature of medical 
record audits. Although all participants were assured of the confidentiality 
of the data and the fact that patient identities would remain anonymous, there 
may well have been concern about allowing sensitive data such as that 
contained in office medical records to be reviewed by individuals who were not 
health care providers in the practice. It is also likely that issues related to 
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malpractice suits, third party reimbursement, and physician relicensure and 
recer tif ication were contributing factors in decisions not to participate in 
this study. 
Data Collection 

As was mentioned in an earlier section, each abstractor was assigned a 
group of practices to visit during the three-month period available for this 
activity. Abstractors were instructed to contact participants by telephone 
and arrange an appointment to visit the practice. The physician^s assistant 
was itiformed of the number of medical records to be abstracted for each medical 
condition and was asked to have these records assembled for the abstractor by 
the time of her appointment. 

Since each physician*s assistant had retained his/her own practice log 
form> it was felt that this form could serve as the basis for retrieving 
appropriate charts. However, it was more often the case that when the 
abstractor arrived at the practice, she found that the indicated records had 
not been pulled. Thus, a half day or more was spent attempting to retrieve 
appropriate charts utilizing record systems that were often not well organized. 
Because of these circumstances, abstractors were often not able to review 
as many medical records as would have otherwise been possible. Moreover, 
in some instances the format cf the records themselves resulted in longer 
periods of time being required for the audit. As soon as the data collection 
at each practice had been completed, the abstracting forms were forwarded 
to the National Board for processing. 

A frequency distribution of the number of charts per medical condition 
for each study participant is provided in Table 16. Audit data from a total 
of 2954 medical records were available for analysis.^ It is interesting to 

^This total represents the records of 116 participants. The records of 18 
physician^s assistants had to be dropped from the analyses because of questions 
concerning the reliability of the data. 
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note that there is considerable variation in the number of medical records 
audited across the vaiious medical conditions. If one can assume that the 
number of medical records audited for each medical condition is a reasonable 
approximation of the relative frequency witM which P.A.'s see patients with these 
medical conditions, the frequency distribution suggests that physician's 
assistants spend more time managing certain medical conditions and very little 
time managing others. For example, the number of records of patients with 
tonsillopharyngit is , otitis media, and hypertension as well as patients being 
seen for health maintenance is far greater than the number of records of patients 
with asthma, angina, or those seeking birth control devices. Although there is 
a fair degree of overlap among participants in terms of the types of medical 
records available for audit, there are a number of medical conditions for which 
there are little available audit data. Moreover, in a number of instances there 
were audit data from only one medical record in relation to a medical condition 
for a given participant. Because it has been our experience with other medical 
record projects that the performance of a given health professional varies 
dramatically across patients being seen for the same medical condition, it 
was felt that the availability of only one medical record for a given medical 
condition would not constitute a sufficiently representative sample of the 
behavior of a physician*s assistant. For this reason, when audit data from 
only one medical record x;ere available for a given condition, they were not 
included in the statistical analyses, (These cases have also been dropped 
from the frequency distribution) .> 
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Table 16 
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It will also be noted from Table 16 that in instances where more than one 
taedical record is available, the actual number of records for a given participant 
may vary from two to as many as thirty for a given medical condition. Because 
of the analytical problems posed by this variation in the number of records 
available for each participant, all scores were standardized prior to the 
statistica] analyses as described in subsequent sections. However, even given 
the standardization process, one needs to be aware of the fact that the number 
of subjects included in the various statistical analyses performed on these 
data is i\ot uniform. 

Scoring and Measurement ^T ssues 

The audit data tor all medical records were treated statistically by 

calculating a comnJiance scoie for each using the following formula: 

Sum of "Yes" entries , 

Sum of ^'Yes" + Sum of "No" entries ^ 

where , 

"Yes" indicates the audit criterion was met 
"No" indicates the criterion was not met 

Because of the variability in the number of records per medical condition 

across participants, all compliance scores were standardized prior to further 

analysis. One ot the most fundamental measurement ^^^'^^^ related to the analysis 

of the data from this study involves the nature of the predictive instrument 

(i.e., the Certifying Examination) and the criterion measure (i.e., audit data). 

While the data on the predictive instrument was the same for all subjects in this 

study, the data on the criterion measure is probably not identical for any two 

suDjects. In addition, it should be remembered that thf: Certifyii^g Examination 

is designed to assess aspects of knowledge and skills which cannot be assessed 

directly by an audit of medical records. 
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In order to provide a sharper focus for the analysis of data between the 
predictive instrument and the criterion measure, an attempt was made to classify 
the data from both sources in relation to what appeared to be logical components 
for analysis. In relation to the Certifying Examination, the test items were 
classified into the following components of competence: 

a. data gathering on patient management problems (PMP's) 

b. management-treatment on patient management problems (PMP's) 

c. interpretation of clinical findings on multiple-choice questions (MCQ's) 

d. management-treatment on multiple-choice questions (MCQ*s) 

Within each of the components of competence listed above, a broad sample 
of medical content and medical conditions is included. In fact, it should be 
pointed out that the sampling of medical conditions included on the Certifying 
Examination is far broader than that which was attempted for the medical record 
audit. To the extent that we know from previous statistical analyses that 
performance on examination materials varies as a function of the medical content 
being sampled, it should be recognized that the medical content of the predictive 
and criterion measures could not be controlled in this study. Therefore the 
magnitude of relationships observed between performance on the Certifying 
Examination and the medical record audit needs to be viewed in this context. 

In relation to the medical record audit, it was felt that the data were 
related to two components of competence: 

a. skill in diagnostic data gathering; and 

b. skill in patient management (including patient counseling and 
ins true tion) . 

However, in reviewing the number of audit criteria related to management 
for each condition, it was determined that the number of items was too small 
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to constitute the meaningful score. Thus, two audit scores were possible 
for each medical condition: 

a. data gathering; and 

b. composite (consisting of data gathering and management criteria). 
These two scores were used to describe performance in relation to most of 
the medical conditions used in the record audit. In some instances, the 
number of data gathering criteria was too small to be treated as a separate 
scoring component. When this occurred, only a composite audit score was 
reported for the medical condition. 

In addition to developing scores for each medical condition, clustered 
scores were developed for some medical conditions which were grouped together. 
Three groups of medical conditions were developed: 

Cluster K : Initial evaluations for headache and hypertension 

Cluster B : Initial evaluations for tonsillopharyngi t is , otitis media 
and health maintenance 

Cluster C : Initial evaluations for tonsillopharyngit is , otitis media, 
urinary tract infection and vaginitis 
The identification of additional clusters was not possible because of the 
relatively infrequent availability of audit data for the remaining medical 
condit ions . 

Findings 

In order to determine whether there was a correlation between compliance 
with audit criteria and performance on the Certifying Examination, Pearson 
product-moment correlation coefficients were calculated for each medical con- 
dition in relation to each examination component. One hundred ten correlation 
coefficients were calculated, and of that number only ten reached significance 
at the .05 level of confidence. There were no significant correlations between 
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compliance with audit criteria and performance on the data gathering or 
ir»anagement components of patient management problems. The only significant 
relationships observed were between compliance scores for some medical conditions 
and the two multiple-choice components. Correlations between vaginitis com- 
pliance scores and both multiple-choice components of the examination ranged 
from .A6 - .52. For venereal disease the correlation between the compliance 
scores and the multiple choice management component equaled .36 and .39. 
Finally, the correlation between compliance scores for the cluster containing 
otitis media, sore throat, urinary tract infection and vaginitis and the two 
multiple-choice components equaled .38 and .46. All coefficients were significant 
at the .05 level of confidence. 

However, in view of the large number of coefficients which were calculated, 
one should view the significance of these ten with some caution. It is generally 
accepted that in calculating a large number of correlation coefficients, some 
will be significant by chance alone. Although the presence of a significant cor- 
relation between a cluster of high-frequency primary care conditions and multiple 
choice scores provides encouraging evidence for the validity of the examination; 
the lack of correlation for other conditions such as headache and health main- 
tenance makes interpretation of the data difficult. Thus, the conclusion most 
consistent with data presented in Table T? is that, in general, the physician's 
assistants included in this study do not demonstrate a similar rank order in 
practice, as reflected by the medical record audit, as they did on the Certifying 
Examination . 

A number of factors may have contributed to the lack of correlation between 
examination performance and compliance with audit criteria. While the medical 
content sampled on the Certifying Examination includes more than 50 different 
medical problems and conditions, the content of the medical record audit 
employed in this study sampled only twelve different medical conditions. It 
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Table 17 

Correlation Coefficients Between Audit Scores 
and Examination Scores 



Audit Data 



MCQ Interpretation 



MCQ Management 



PMP/Data Gathering 



PMP Management 



Angina 

Data gathering 
Composite 



-.09(12) 
-.03 



,05 
.03 



,45 
,41 



32 



Asthma 

Data gathering 
Compos ite 



•.35(15) 
-.37 



.20 
,23 



-.25 
-.26 



.11 



Birth Control Pills 
Data gathering 
Composite 

Diabetes Mellitus 
Data gathering 
Compos ite 



.87*(7) 
.79* 



-.40(9) 
-.40 



.44 
.46 



.09 
.04 



-.03 
.04 



,33 
25 



.62 



.59 



I 
I 



Headache 

Data gathering 
Compos ite 



-.09(55) 
-.09 



.03 
.03 



.17 
.17 



-.07 



H. Maintenance 
Data gathering 
Composite 



-.13(65) 
-.10 



.12 
.13 



.12 
.14 



.15 



Hypertension 
Data gathering 
Composite 



-.08(33) 
-.09 



-.01 
-.08 



.05 
.06 



.00 



Otitis Media 
Composite 



.04(83) 



-.07 



.18 



-.07 



103 



Sore Throat 
Composi te 



-.12(102) 



-.13 



.04 



.14 



*indicates significance at the .05 level 

( ) numbers in parentheses indicate the number of cases available for analysis 

— no correlation coefficients were calculated between data gathering and management data sets 
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Audit Data 

Urinary Tract 
Infection 

Data gathering 

Composite 

Vaginitis 

Data gathering 
Composite 

Venereal Disease 
Management 
Composl te 

Diabetes Mellltus 
(Follow-up) 
Composite 

Diastolic Hypertension 
(Follow-up) 
Composite 

As t hma (Foil ow- up ) 
Composite 

Birth Control Pills 
(Follow-up) 
Composite 

Angina (Follow-up) 
Compos! te 

Headache + Hypertension 

H. Maintenance *f Otitis 
Media + Sore Throat 

Otitis Media + Sore 
Throat + UTI + Vaginitis 
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MCQ Interpretati on 



-.18(65) 
-.10 

.A6(39)* 
.46 

.06(36) 
-.11 

.15(37) 

.03(76) 
.01(20) 

.07(16) 

.28(22) 
-.07(21) 

-.10(49) 
.38(26)* 



MCQ Management PMP/Pata Gathering PMP Management 

-.10 -.01 

-.02 .04 -.04 

.48* .27 

.52* .27 .14 

.39* ~ -.08 

.36* -.06 -.14 

-.1^ -.13 -.13 

-.12 .13 -.06 

-.13 .07 .00 

-.18 -.31 -.48 

-.34 -.30 -.23 

-.07 .21 -.03 

-•21 .24 .01 

.^6* .05 .08 IQQ 
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had been anticipated that this inability to control for the content in these 
two sources of data would make it difficult to evaluate the effectiveness of 
the Certifying Examination as a predictor of subsequent practice performance. 

Previous evidence of the influence of medical content on performance had 
been obtained from the analyses of several certifying examinations. In an effort 
to determine whether medical content was also a factor in the extent of com- 
pliance with audit criteria, correlation coefficients were calculated among 
the compliance scores for the medical conditions included in this study* The 
resulting inter-correlation matrix is presented in Table 18. 

One hundred eighteen correlation coefficients were calculated. Of that 
number, 77 (or 66%) did not reach significance at the .05 level of confidence. 
Of the Al coefficients that did reach statistical significance, only 2A 
involved sample sites of twenty or more cases. These coefficients ranged in 
magninude from .31 to .78. Thus, there appears to be considerable variation in 
compliance with audit criteria across medical conditions. The fact that content 
appears to be a significant factor in determining performance both on examinations 
and on record audits poses significant problems for accurately assessing the 
true relationships between these two measures. 

Another factor to be considered is that although medical record audits are 
coming to be widely used» there is not yet a consistent body of literature which 
supports the validity of this assessment technique. Although it seems unlikely 
that health professionals would systematically fabricate entries in the medical 
record* it seems reasonable to suspect that they do not document all pertinent 
behavior. Tl^us , what appears to be lack of compliance with audit criteria may 
simply be a failure to record all pertinent aspects of actual behavior. If 
such Is the case* jt raises a number of important questions concerning the 
validity of medica'' record audits. 
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-.16 
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(6) 


(3) 


7 


* .93 


.87 


-.37 


* .97 




(5) 


(A) 


(3) 


(6) 


8 


-.01 


* .73 


-.46 


97 




(8) 


(12) 


(7) 


(3) 


9 


. 39 


* .68 


-.03 


.06 




(10) 


(14) 


(6) 


(5) 


10 


.46 


* .78 


.01 


--.71 




(7) 


(10) 


(A) 


(4) 


11 


.47 


* .73 


.07 
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(6) 




12 


.93 


.55 








(3) 


(8) 






13 


.64 




. 13 


. 13 




(4) 




(4) 


(7) 


14 


.34 


.98 


-.21 


14 




(9) 


(6) 


(6) 


(9) 


15 


.97 










(3) 








16 






-.68 
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17 


-.32 
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(35) 








.54 


.46 






(11) 


(]C) 






.37 


.44 


-.05 




(9) 


(12) 


(4) 




.09 


.37 


.47 


-.35 


(18) 


(21) 


(10) 


(6) 



Indicates that less than 3 cases were available for analysis 

( ) Indicates the number of cases used in calculating the correlation coefficient 

* Indicates a correlation coefficient involving less than 20 cases, but significant at the .05 level of confidence i 

Indicates a correlation coefficient involving more than 20 cases, and significant at the .05 level of confidence \ 
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If, however, medical records can be assumed to accurately reflect the 
behavior of health professionals, then factors in addition to variability in 
content must be considered in attempting to explain the lack of correlation 
between examination performance and compliance with audit criteria. 

One possible explanation is that what physician's assistants know about 
the diagnostic evaluation and management of patients and what they actually do 
in their respective practice environments may be significantly different. If 
their motivation in taking an examination is to demonstrate that they know 
what constitutes optimum care, their motivation in a practice setting may be 
to conform to patterns of health care delivery and record keeping which meet 
the expectations of the physician employer. If the resulting behavior is 
different in the practice setting from what it is in the testing situation, one 
cannot expect to observe significant positive correlations between the two. 

If the differences in performance do exist between testing situations 
and practice settings and if medical record audits are valid assessments of 
behavior, then it may be unreasonable to expect that physician's assistants will 
fall in the same rank order on a medical audit as they do on a Certifying 
Examination. 

Since a certifying examination seeks only to identify individuals who 
have not achieved miniroum acceptable standards, then perhaps the same rationale 
should be applied to the assessment of co^npliance with audit criteria. If 
this viewpoint is accepted, then one would hope that individuals classified as 
acceptable or unacceptable in terms of their examination performance would be 
similarly classified as acceptable or unacceptable in terms of their compliance 
with minimum standards of care as reflected in a medical record audits. 

The pass-fail level on the 19?A Certifying Examination was set at a 
standard score of 420 or sixty-four percent correct. For the purposes of 
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this comparison between pass-fail status on the examination and pass-fail 
status in relation to compliance with audit criteria, a sixty-four percent 
compliance rate was set for the record audit. A two-way contingency table 
was used to tally each physician assistant's pass-fail designation based 
upon his examination score and his audit compliance score. A chi square 
analysis was applied to these data as displayed in Table 19. The chi square 
value equaled zero, indicating that there was no relationship between pass-fail 
status on the examination and pass-fail status in terms of audit compliance. 

As can be observed from Table 19, a rather large number of the participants 
(30) passed the examination but failed to meet the minimum audit compliance 
level. An inspection of the examination scores of this group of participants 
indicated that although many of them did achieve a passing score on the 
examlna tioPi, the score was near the pass-fail level. For this reason, a new 
contingency table was prepared, as illustrated in Table 20, and the pass-fail 
level for the examin^^tlon and audit compliance was adjusted upward from sixty- 
four percent to sixty-aine percent. The resulting chi square equaled 5.47 
which was significant at the .02 level of confidence, thus indicating a signi- 
ficant relationship between pass-fail status on the Ceitifying Examination and 
pass-fail in relar.ion to compliance with audit criteria. 

Conclusions and Recomm e nd at ions 

On the surface, the results of this study would seem to indicate that 
the rank order of physician* s assistants in terms of their compliance with 
audit ci'iterla does not correlate significantly with their rank order in 
terms of scores on the Certifying Examination. However, in relation to 
minimum acceptable standards, a significant relationship appears to exist 
between the Certifying Examination and compliance with audit criteria when 
the minimum standard for both is set at 69 percent. Finally, there appears 
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Table 19 

Contingency Table 
Minimum Standards 

EXAMINATION (64%) 



Pass 



Fail 



A 
U 
D 
I 
T 

(64Z) 



Pass 



Fail 



72 


10 

i 


30 





» 0.0 
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Table 20 



Contingency Table 
Revised Minimum Standards 



Examination (69%)* 







Pass 


Fail 


A 




■ 




U 


Pass 


39 


21 


D 








I 
T 

(69Z) 


Fail 


26 


! 30 

i 
1 

1 



x2 - 5.A7 (p « .02) 



*lf this standard had been applied to actual 
examination data, it would have resulted in 
a 50% failure rate. 
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1975 PHYSICIAN'S ASSISTANT CERTIHING EXAMINATION 



The third Certifying Examination was administered to 1411 candidates 
in 56 test centers on November 19-21, 1975. The content and format of the 
examination were equivalent to the 197A examination program. The one-day 
written examination was divided into two major components: multiple- 
choice questions and patient management problems. In addition to the 
written portion of the examination, each candidate underwent assessment 
of his/her skill in performing selected components of a physical 
examination. 

Items on the multiple-choice question section of the examination 
were designed to assess the candidate's knowledge and his/her skill 
in applying knowledge related to high priority health care functions. 
These items covered materials in the following broad areas of competence. 

a. selection of laboratory and diagnostic procedures; 

b. interpretation of clinical findings; 
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c. selection of management strategies; 

d. patient counseling and instruction; 

e. knowledge of medical sciences; and 

f. knowledge of behavioral sciences. 

Items on the patient management problems were classified into two subscales 
data gathering and management /therapy. Candidates were presented with 13 
simulated cases and asked to make decisions regarding the appropriate diagnostic 
workup and management of the patient as would be done in an actual clinical 
setting. Thes^ problems were designed to assess the candidate's skill in 
gathering pertinent information about patients and in making appropriate 
management decisions. The Certifying Examination has continued to focus on 
primary care» and for this reason the content of the examination was drawn 
from all of the clinical disciplines. 

The assessment of physical examination skills continued to focus on each 
candidate's ability to examine the heart, lungs, eyes, abdomen, and neurologic 
syste'j. As was the case in 1974, each candidate was evaluated by two physician 
examiners, each of which observed the candidate performing different components 
of the physical examination. 

Table 21 summarizes the composition of the candidate group in 1975, 
There were no marked changes in the proportion of candidates from each of 
the four categories* Seventy-three percent of the candidates received their 
training in physician's assistant programs, eleven percent in Medex programs, 
and five percent in nurse practitioner programs. Eleven percent of the 
candidates qualified for the examination on the basis of work experierice 
(informal training) • 
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Table 21, 

Coraposition of Candidate Group 
1975 Examination 



Type of Training Number Percent 

Physician's Assistant 1028 73 

Medex 151 11 

Nurse Practitioner 80 5 

Informally Trained 152 11 

Total lAll 100 



For scoring purposes, four subsections of the examination were identified: 
MCQ : represents the multiple-choice question portion of the examination. 
PMP (D/G) : represents those sections of the patient management problems 
that involved the selection of historical questions and physical examination 
procedures . 

PMP {M/T) : represents those sections of the patient mangement problems that 
involved the selection of laboratory studies and management decisions. 
PAS ; repriasents the physical examination skill assessment portion of the 
examination. (All examination components - heart, lungs, eyes, abdomen 
and neurologic - were weighted equally) . 

As shown in Table 22, the reliability of the total examination was .92, 
which places it within the range of reliabilities for other National Board 
examinations. The mean difficulty level for the multiple-choice (MCQ) section 
of the examination equaled .58 which indicates a slightly more difficult set 
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of questions coropared to other National Board examinations, T\\is statistic 
indicates that the average candidate from a formal training program answered 
about 58 percent of the multiple-choice questions correctly. The mean 
difficulty level of the patient management problems (PMP) was .79 for the 

Manageinent Therapy section and .77 for the Data-Gathering section. These 
-"difficulty levels correspond to those found on other PMP examinations. On the 

Performance Assessment (PAS) portion of the examination, the average candidate 

from a formal training program obtained 75 percent cf the total possible points. 

This represents an improvement in performance over that observed on last year's 

examination where the average candidate obtained 67 percent of the total 

possible points . 

Table 22 

Examination Statistics 
1975 Examination 



Statistic 

Composite Reliability .92 
Mean Difficulty 

MCQ . .58 

PJfP (D/G) .77 

PMP (M/T) .79 

PAS .75 
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The statistical analysis of the 1975 examination indicated that the 
composite reliability and difficulty levels were comparable to those observed 
on the 1974 examination ♦ The improved performance on the physical examination 
assessment as noted above was probably due to better candidate preparation 
for this portion of the Certifying Examination^ 

Table 23 summarizes the examination statistics that have been obtained during 
the past three years ♦ The statistics were calculated on the reference group which, 
for the past two years, has been defined as candidates who are graduates of formal 
training programs and are taking the examination for the first time^ The column 
labeled "average P" refers to the difficulty level of the examination and describes 
the proportion of items answered correctly by an average candidate. Reliability 
estimates have been calculated using the Ruder-Richardson 20 formula which is 
an estimate of internal consistency. 
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Table 23 

1975 - Physjician's Assistant Examination 

I 

Performance of the Eramlnation 
(Ref«ffnce Group) 



Ite« Analysis Data 



A. ttotal Test 



Year 

1973 
1974 
1975 



Nuttbcr 
of Items 

606 

1266 

1215 



Composite 
Reliability 

.93 
.92 



Year 

1973 
1974 
19V5 



Number 
of Items 

147 

216 

210 



Average P 
.64 
.57 
.58 



Reliability 
.83 
.89 
.87 



C. (PKP (Hg'i>t . /T herapy)! 



Year 

1973 
1974 
1975 



Humber 
of Itema 

3? 

284 

289 



Average P 
*82 
*78 
.79 



Reliability 
,54 
.76 
.69 



(PMP (Data /Cath.)i 



Year 
1973 
1974 
1975 



Number 
of Items 

422 

554 

489 



Average P 
.79 
.75 
.77 



Reliability 
.80 
.89 
.88 



E. jPASl (Performance Assessment) 



Year 

1974 
1975 



Number 
of Items 

212 

227 



Average P 
.67 
.75 



Reliability 
.91 
.9^ 
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Table 24 summarizes the intercorrelat ions among the various components of 
the 1975 Certifying Examination^ These relationships are consistent with 
those observed on the two prior examinations, and suggest that each of the 
examination components is assessing aspects of competence that are different 
from chose .assessed by other components' of the examination^ 

Table 24 

Intercorrelation matrix of subscores 
1975 Examination*** 



(Performance 

PMP (Mgmt> /Therapy) PHP (Data/GathQ PAS Assessment) 
MCq ,57 (-73) .37 (-42) .27 (.29) 

P}IP (Mgmc. /Therapy) .32 (-41) .17 (-21) 

PMP (Data/Cath.) .18 (.19) 



***Correlat ions in parentheses have been corrected for unreliability. 

Table 25 sumnarizes the performance by candidate group on the 1974 and 1975 
Certifying Examlnat ions^ The composite score is derived by assigning equal 
weights to all components of the examination* Thus, each component contributes 
25X to the total test score^ It will be noted that candidates taking the 
examination for the first time score consistently and significantly higher 
than candidates who are repeating the examination ♦ The informally trained 
candidates score consistently lower than candidates who are graduates of 
formal educational programs, and their performance on the 1975 examination 
declined from their level of performance in 1974* As might be expected, the 
failure rates for inforxnelly trained candidates and those who are repeating 
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the examination are considerably higher than failure rates for other groups 
of candidates. The nurse practitioners who registered for this examination 
continued to perform well, although the.y are too few In number to permit any 
generalization to the performance of nurse practitioners as a whole. 
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Nrformwict on rby«l<il«ii'» Aisittitat Ctrtifylnt SxanlMtlcm 



A* Ptrformance on T otal Tet t^ 



1974 
1975 



Reference 
Group 

500 70 
500 70 



First 
Takers 

T~s7d. 



491 
489 



75 
78 



B» Performance on MCQ 



Reference 

Group 

T sTd7 



First 
Takers 
1 S.D. 



1974 
1975 



500 100 
500 100 



490 
^88 



105 
107 



C. Performance on PMP - HAmtwTrt^ 



Reference 

Group 

T sTdT 



First 
Takers 

\ — sTd. 



1974 
1975 



500 
500 



lOi 
100 



496 
492 



103 
104 



D. Perform*nce on PHP - D/G 



Reference 

Group 

"7 s7d» 



First 
Takers 



1974 
1975 



500 
500 



100 
100 



494 
492 



103 
105 



£• PerformAnce on PAS Composite 



Reference 
Group 



1974^ 
1975^ 



500 
500 



100 
100 




Re^esters 



442 
384 



54 

7/ 



Repeaters 

T sTdT 



390 
345 



78 
98 



Repeaters 

T S>D> 

448 89 
372 139 



Repeaters 

T s7d7 



467 
423 



79 
97 



Repeaters 



464 
398 



100 
137 



Informally 
Trained 

T sToT 



426 
387 



83 
87 



Informally 
Trained 

'^TT s.D. 



414 
375 



110 
110 



Informally 
Trained 



461 
408 



114 
118 



Informally 
Trained 



450 
409 



112 
112 



Informally 
Trained 
T S>D> 



379 
358 



126 
142 



460 
476 



73 
86 



Modex 
t S>D> 



419 
430 



97 
112 



Medex 



476 
A64 



118 
135 



Medex 



485 
497 



102 
100 



>tedex 
T S,D> 

460 103 
512 104 



503 
493 



68 

72 



510 95 
496 103 



P. A. 



501 96 
494 103 



P,A 



499 
494 



100 
101 



504 100 
492 102 



Nurses 
T S.D, 



499 

521 



96 
70 



Hura es 

S.D* 



488 
528 



116 
96 



Nurses 



530 
528 



S>D> 

98 
89 



Nurses 



502 
517 



105 
107 



Nurses 
T S.D> 

472 151 
514 104 



Total 

Orov)p 

"t sTd 



490 75 
461 83 



ToUl 
Group 

t sTd. 



487 105 
478 113 



Total 

Grout. 



494 103 
483 112 




102 
106 




jThe composite score for the total test - JT (PMP - MCKT./Ther.) + 1 (PMP - Data/Gath.) + 1 (MCQ) + 1 (PAsJ /4 
The PAS Composite for 1974 was weighted affording to the length of each Individual section. 

^In 1975 > each section of the PAS vaa standardized before the composite vas computed. The PAS Covposlte was 
then restandardlzed to have a mean of 500 and a standard deviation of 100 after the 5 sections were added together. 
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I 

The graduates of Medex programs performed as veil as P. A, program graduates 
on the data gathering sections' of patient management problems. In 1974, they 
scox"ed significantly lower on the physical examination assessment component, 
but their performance was significantly better In 1975, In relation to the 
PMP management /treatment and multiple-choice question sections of the examination, 
the Medex graduates scored significantly lower than the graduates of P. A. programs* 
Similar score comparisons were also observed in 1974, 

The examination standard set in 1974 and again in 1975 equaled a composite 
score of 420. At this composite score, the overall failure rat;e equaled twenty 
percent* However, as might be expected from the differences in examination 
performance, the failure rates varied across the several candidate groups. 
For example. In 19 75, the failure rate for Informally trained candidates 
equaled 60. 5X, while the failure rate for Medex program graduates equaled 23. 
and for P. A* program graduates, 14. The failure rate for the nurce practitioner 
group equaled 7.5X. 

Beginning with the 1976 Certifying Examination, a certain proportion of 
the test will be drawn from the pool of previously used test items. Moreover, 
the number of test committees will be reduced from 4 to 2, one with 
responsibility for the patient management problems and the other with 
responsibility for the multiple--choice questions. Both committees will continue 
to be In ter-dlsclpllnary , including physicians, nurses, and physician's assistants* 
It is anticipated that these measures will not only streamline development of 
the examination itself, but enhance Its cost effectiveness. 
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At the present time, the pool of multiple-choice questions totals 572 test 
items. They are distributed across clinical disciples in the following numbers: 

Items 

Medicine 248 
Pediatrics 121 
Surgery 88 
Obstetrics-Gynecology 36 
Behavioral sciences 79 

In addition to the multiple-choice questions, the pool contains 38 patient 
management problems distributed across the clinical disciplines as follows: 

Patient Management Problems 
Medicine 14 
Pediatrics 15 
Surgery 2 
Obstetrics-Gynecology 5 
Behavioral sciences 2 
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APPENDICES 
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APPENDIX A 

-89- 
CASE D 



Mrs. Arlene Burns is 34 years old. She is a new patient and has brought with her a 
summary of her medical history which was provided by her physician in another state. 
Her history is unremarkable except for the presence of recurring migraine headaches 
during the past 18 years. 

In response to your inquiry concerning why she is coming to the office today, Mrs. Bums 
responds, "My migraines have been acting up lately and Td like to get my prescriptions 
renewed. You'll find everything in my record. Tve been taking Valium, codeine, and 
Cafergot." 



TUBN IMMEDIATELY TO PAGE 4. 



128 



-90- 



SELECT ONLY C»ra:. 



205. What's happening In your life that's 
stressing you? 



205. Nothing in mrttcuUr. Aa :i matter 
of fact, we have )u8t movt,Ki into a 
n*/w hou3o. The community i3 much 
better thait the one we had been Hvlnfc 
in b<^tore movini^ 1500 niUes to here. 
My husband' a job couldn't bo hotter 
• n t»*:'ni3 of n:oney ?j\d poaition. . . 
thai'3 whv '.vo came here in rhe firac 

{'rjHN rmiE diate ly to 

PACE n)* 



206. rd be glad to renew your 

prescriptions, but first let's 
go over how often you need 
to take them. 



206. Of.ay, v/iul do you wr^u i^j knew? 

(TUim IMMKOUTEf.V TO ?aQE 7)* 



207. Your migraines are acting up? 



207. Ve«, for th*? ['\3i tow montha or 3o 
U joorr.n Uito L\cy nave boen ^^t^ttincc 
.i^rt» frt^n»u»nt than U3ual. (TUHN 
:Mr.:EDIATKL\' TO PACK e)* 



208. rd like to check you first before 
renewing your prescriptions. 



208. v/"ll. okay, if vo*j ..t\) Aeco^Mirv 
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SELECT ONLY ONE. 

209. Have you had any dlxtinesa or any 209. J^'^t fh^ ^r^m** Mi -phlnS. 



7 



symptoma other than your migraines? }U3t aro occurring more oiu a, i^iac'a 

nil. (run:; immkdiately to 



210. Do your current headachea feel any 
different than your uauai migraines? 



210. No, th^r dnrr.o .:>id caina. T^un- 
i'jet n?>5 occurrin.T moni oit<jn. tlint'3 

(TURN IMMiUL^'ATH LY TO 
1' ^f rK 



211 . How often have you needed to take 
your medication? 



211. t^lciy, it de«ma like I azn i:(k\a^ 

Valinm almost daily. ■ . about 4 <L<;ca ? 
iny ior the laat months, Tl^e oth#»r 
.T.tjdicinita, only xvh*;n ( need tiinm, 
^'n^ch recently haj »»eu for tvc neadnchoj 
.1 week. ConornUy. I Jtick to nut iriKmj 
-i^TT ;C C^;u'r;.'<^i ^iISh a week. The laai 

,^o«;n out of the codeine for about t".vc 
wtr..«k:^, but I Jv)n't tuiitf it •xcept rarely 
Vf*»*aujt; I i;cl rficiC to my atomach. 
[TimU IMNfEDLMELY TO rAGS 10)* 



212. Could you explain what you mean 
when you say your migraines are 
acting up ? 



212. V'^a, Tor the laat ^fv/ or .^o it 

,\}ema Uk<^ tht»y hnvt* be**n g»it(my 
ii'ooutiut ainn ti*tini. (TURN 
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SELECT ONLY ONE. 



213. How often have you needed to take 
your medicine? 



213. uiteiv, it seems like I am takiag Uio 
. ;;.Ui.i s.\::.zzt ^?.ily. .» about t tinK-a 
3 aav for the Ipst few months. The 
'jiher cnly when I <^ 

vjiirh rocentiv has been lor a/:; lior.cl- 
nchofi n week. Jeneraiiy, 1 siiek ic 
/:i trJui^i^ c^ci 10 CaJergot vAW^ k 
• <=^^K. The \'^^\ few \vc*eks r\L- i^cne 
■ «-» b'^cn cvit cf the 

^ccleir\e lor nJxui nvo weeks, l?ui T 
con'i t::J:c it e.xocnt rarely becau^je 
I get .«!lck to ny stonric!) (Tri\N^ 
I M \\ E DL^ ;:: L V T O ^ A G i: 1 0) - 



214. What's happened In your life that's 
stressing you? 



214. 



I act, we havc just moved cito r u^v.' 
..ouwc T:;c ccmniunity n-.-ch better 
'.ui^ lijs: hnd Lvffii MV\n? m refer 

. h.usband*s job ccuicn i 

. . . triors whv rrme here m> the first 



215. What Is the usual pattern lor your 
headaches and how has it been 
different lately? 



215. 



216. Are you taking your medicine 
correctly? 



216. 
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SELECT ONLY ONE. 



217. Do you feel you are alipping up 
occaalonally and^ perhapa^ going 
over the required dose? 



217. : ;:r.ov; bv now how lo take my 

mricucationa. God knows i'^-: .-.^u 
. 'Jiing "'irtia /or ciluiojt 20 yonr^' 
(TV:ill IMMEDIATELY TO PAGE \^)' 



218. Are you taking one medication more 
heavily than the others ? 



218. No, not r?:\lly. IV s jmt th-^t 

'TURN IMMEDLMELY TO ?AG3 13'/^ 



219. Do you find you need to increase 
your dose In order to get the same 
effect? 



219. ::c, not ronlly. IV^ Just t^-' I'-n 
^octin^ :iio iicadciciiL.: more 
.■-CJt-nUy. (TURN (MNrr:nTATr:L: 



220. How long have you needed to take 
your medication? 



; ..nv '* --^V' I'Hrt ru'A- injnil:i3. llie 
\:\ } ""^'i-j'vj l''".r '/^'.^ wool's, ':\r 

.:.rMc::;:^\ r:i: ;.v ro r :?) * 
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SELECT ONLY ONE. 

221. Have you and your husband had 221. Ail I cauv^ m iiere luv u/;^., -v. ...^ 

satisfactory reJationshlps lately? y. '<i:cri::uci\£> lenewed for a concUicn 

Tve hnd fcr 20 voara ana vov; kotp 
'j^.of^oriii!^ with quosr.ions . ^^'^ji^.h i 
ycu }ust c^v?ck my records atvJ 

n^edicmo? ('i UHN IMMEDL^TILLY 
TO F/.C:i 



222. Well, It certainly doesn't sound like 
stress from what you describe, so It 
probably would be a good Idea to go 
Into more careful detail on your 
headaches. What is the usual 
pattern for your headaches and how 
has it differed recently? 



222. Wornuuiy, I might have n ^f-'^dncia- 

i-very 3 months or so, bi:t fcr ^ne lr.3t 
= '^.or.thn, it':: been iacn'-asi^!;^ *o 
" :o I'cini ^h'^i ii'^ avirnf^mc; one unit: 
vr weo.^'. . CijK^r^'iPe, the horiudcn^j.-a 
:ccl U\Q namo. {TUPN IMMEDIATELY 



223. It sounds like thereWe been a lot 
of changes In your life. That 
could certainly be unsettling. 



223. '-ertn-.r.iy -^i - .:cs,^rciIitA' 

'because lately I've Mit^a qui^^ 
'^y^s^^ With '^'V^^ ryonc . { i "^,.^N 

i:':Mii:DL-MELY TO r/.G?: \-\] 



224. Sounds like it could be a carry over 
from the work of moving. What do 
you think? 



224. mac cc-rtamiy is a posi?ioility 
>»'rnu.^e irtf*iv i>-en cu;te 
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SELECT ONLY ONE. 



225. Even though your headaches may 
feel the aame to you, I think It - 
would be a good Idea to have a 
neurologist check you to make 
sure Uiat iVs nothing more serious, 
Ho^ does that sound to you? 



225. Well, if 'JnCa what you UUak ia bnjat, 
a^t it up ad quickly aa poaaiblc, 5o 
: cnn g:t ray medicine ronewed. 
{Tm3 vi tho f^Ttd of *)\e cnae, TURN 
IMT.IKDL^THLy TO PAGE 31) ♦ 



.V 



228. What Is the usual pattern for your 
headaches and how has It been 
different recently? 



226. 'oraj uly, { av^tt nnve a ne.A<n'\cne 
*\":rv 3 monthg or Jo, but for U4vi 

— ^•'>|; C^\h*jnvi:^e, the 
h-aaachoa ivM tl)0 vinmo* (TUP.N 

^^^Iv^^^:DL^T^LY to ?ag£: i i; * 



227. Could you describe a typical headache? 



227. .Ui I cHm*j In haro for woa to rft*t my ^ 
preac«'^DtionH r*^nHW4>c3 for n conUUion 
V\<i U\(\ for ?0 yo*^ri.< m^i yovi ktjop 
badcr^jrmcr me with quuatlona. Couldn't 
vou jurit checK my r«corci3 and ronew 
rn- mt^d^rlno? (TURN IMMKOUTgLY 
TO VA^Jii 15)' 



228. Can we go over what in your life 
may be diatreaaing you? 



228. ''otnui^;^ m parlirulnr, as a mntt*ir 
01 fact, wo have juat moved into a 

house. The communit)- \s much 
oeUer than the on& we had bo»*n livinfj 

iol, couldn*t bu better m t>>rnia 'ji 
( .oiicy :^nd poaUion. . tarit <^ ^^ily v,x* 

I'..Mr:DLATn:LY TO PAGE 1)* 
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229. Sounds like something in your life 
may be bothering you. 



SELECT ONLY ONE. 

229. ; 



iNulh;uf> In f:«.2 ticular. a niattor 
of fact, 'X'd havf> juflt'movcd into a 
new house. The community is much 
better than the one we had baew ilvir*^ 
r. 'x foro nK^vmg here. My huubrnd*3 
yah ccuidn't Ic hotter in ter*^*' ' 
n-jonev and j:c<:»t:cn ..U)at'« "-^v n^o 
/\'>rne iiQiv \i\ Cic first place. '.TURN 

::.jMircL\T::LV to pack e)* 



230. Taking that much Vtlium sounds like 
you may be getting yourself into other 
problems. 



230. ..*4 ; came )i) i;oto for ^A"t9 to ^ct ^r.y 
•:rescnntion« rone woe lor a conuiuori 
Vv<; hac for 20 yuariJ and you \^M^-y 
'-\{ur^rxi^T r^^e with Ciueetioiifl. Couldn't 
"-m Jvi£$l cneck toy r*vv)rc3 aivl r::n«H> 



231. Certainly does sound like your 
migraines are acting upl What is 
the usual pattern for your headaches 
and how has it differed recently? 



231. M'^rTYially, I m\^hr h:wo ?. headacht? 
( vcrv :J mcnUis or 3o. uuc tor u ^ 
*'^>^t b m on the u s ceon uicr*jayin*4 
t*,*- .j^p.t it* a avi;r:it:\nf; tne 

l.c.idrirhes itol ihe :jame. (TURN 
7>lMFrMATVt,v TO PAGE 11)' 



232. Why are you taking so much Valium? 



232. '^H J r-^nvj r *--rr' '>r ^ • to Zi't 
'0 tind for ;,*C vcars .-^nd vou kocc 
vi'Af -u:** inv,* rvicc*: -J r.nd j v 
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SELECT ONLY (»nB. 
233. What makes them better or worse? 233. 



: 1 for tne '^,st lew , 

./.r''>«>, { (rv r^^* -..h'O ■'«'<tn 

,a : ; .aJC fOsV Wcn^kd r >'C .;.>yot. .■>^. ...... 



234. rd like to grlve you a physical 
examination to rule out serious 
causes for your headaches. If nothing 
turns up, I think we might want to start 
thinking about having you talk with 
Dr. Cray, our psychiatrist, to see U 
he mii^ht be able to help. Does that 
sound like a reasonable plan? 



234. ..'-^ii, 11 that'a u-nru vou Jiii\i< b>-\U. 
J ciuj <-a'j of cho case. TIT:?' 



235. That's quite an increase in your 
symptoms. What do you think is 
making them more frequent? 



236. ril do the physical examination now. 
If we don*t find anything, we'll renew 
your prescriptions. The physical wir 
tell us if anything more serious mlgh 
be going on. 



235. , i;j r .:.;.,.v. • r^-- 

.'*v) - ,j / • ^l:,/itlv 

.... .'^^i^or: bocauj'3 iiv.?^:"*!? ^^^Irc.j 



236. 



1 I 
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237. rm sorry If I upset you 

like to determine U how often you need 
to take your medicine* 
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SELECT ONLY ONE 
What I would 237. 



Valium ilally. . .nhont 4 tim»?>i 

a day for Witt Ir^t f*:w -noniha. TI^^* 
vj^er m*-.*.*.* j"-*-. I r:*'**:^ 

two h<fau>i<*itttrj vv'tivk. iy 

zone ov^i * l..>iv. 
'o'lo'nf} tor Jcc. :*w 

•To^ f'j^o^ to ••t-^n'ic.c?' > ' * ' 



; \voelc*^j but I 



238. You* re angry with me for what Tve 
said? 



238. ? i^c'Hfi I n Utile, u eccme iil:e 
Tvo been more on edgo lately wUh 
ovor/cr,e. (TUKN IMMEDIATELY 
rC PAGE ) 



239. rm just checking. After all you 
and I are both human and can make 
errors. 



239, Ali I rr,Tne in ht^re ^CT vr-ts to fM ni7 
,:rN.:srript^onF rvr,<;u'nr^ for a ccncit^( r. 
Tve had fcr UO y»iara and you ^:c«t-> 
;-Jjrerinc mf with v;ue8tion5, Ccuidr.'* 

jujn chock nw recordn a>nd ri*new 
:nv n;oilic)ue? ( lUHN LMMEDIATH:/' 
':0 PAGi: 15)* 



240. You seem upset. Do you feel more 
on edge lately? 



240. ! "knio in horv :rr -vui^ ic r nr; 
;r»*fci:npti'nif» r«-;iiewt;0 f'">r ^ in- 1 Men 
h:"-* ff^- '00 yt^:4rfl tint you .%oi.'p 



137 



I I 



-99- 



) 



SELECT ONLY ONE. 



241 . You aeem upaet with me for susking 
that question. 



241. [ r.r.; a little. It sc-eme like 

rvo been more on edj^c lately vmn 
-vcrycnc. (TOTN nvtMEDLJ^TELY TC 

tag:-: 17)* 



242. What's happening in your life that's 
stressing 3rou? 



242. Ncthinr :r. particular. Afc^ u matter oi 
*:ict, ;vti ijavv ju6t moved ir.lo a new 
houfc't:. '**l:t: corr.nvunity if? niuoh better 
Uuin the one v;u iiad bc-^n Ir.Mn*^ in befo»-«* 
ir.ovir.;; iiore, Mv husbanc'a joo couldn't 
fct^uer m Jf*rm6 of money and position 
.. , ihhCf\ we came h^Tfi in Uie first 

:::rvce. (Tini':: mvr^viA'n. i.y to 



243. What is the usual pattern for your 
headaches and how has it been 
different recently? 



243. ..G; ily, ! ::v^^ht ^ Iu:ucache 

..p: t^-vt it't? :lv^'rn^;^^:^; > 

f;BU)*r-. (TUPN IMMEDIATZLV TC 



244. Do you think you're taking your 
medicine correctly? 



244. I should know by now ho^x* to niv 
^'ipdirntions. God knowt; I've been 
L::icinc thc.T. for P-lmoHt 20 yt^nre' 
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245. Valium may not be quite what you 

need. I'd like to give you aomeihing 
to help lift your mood too. Why don't 
I talk it over with the doctor, get 
you the prescriptions, and see you 
again when these headaches better 



-100- 
SELECT ONLY ONE 
245. 



ThHt's fine. Perhaps I can talk 
bett'^r when I [jet these headache.^ 
unJer control. fThis is the end of 
the rnso. T^/v?N fMTylEDLXTi: 



H6. Why cion't we see If we can find 
out why you may be so sensitive 
lately? 



246. :A\ \ canui la here for wis r:) 'j:?c niv 
prose rrpnr^na rcr.nv/cjd for a conauio,; 
I'vv? r.-;c: :jr y^aro and you keep 
\ad^-7rln.q; r.^G vvlLh questions. Coul lu' 



/C- .u3t checl: mv recur.i^j 



7 :n^Jicii\e? (TURN I^:^]ii:DL^T^ LY 



247. Moving certainly is a stressful period 
in one*8 life. Let me renew your 
medications and If you continue to 
have problems, please let me know. 



247. 



t c J <- ■ 



248. With everyone? 



248. d Lj:r.t .;v, )'J!C i ^.UUda .c'.l 

7 . L ,1 J*. J I : \ 1 1 1' : J ;i vnv pj t e r ^ve :\ \ t: 
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SELECT ONLY(»rt:. 

249. I apologize for angering you. I ■ 249. — r rrhir.<5 f car. tetter 

really do feel U would be safer If -^.-n 1 i,et these headaches MrAer 

I went over your symptoms first toiurol. (This is the end of the case, 

in case anything new may have 
occurred. We can do this next 
week, If you like, after I renew 
your medications and things may 
have calmed down a bit. 



250. f'm afraid I couldn't do that until 
we have gone over your case. It 
could very well be something new 
and harmful to you. What has the 
pattern of your headaches been? 



250. Norniaily, I migiU have a heaoache 

' \<.4V J .';iunl!;s or eo, but ioc U\e last 

r:cr::J^e btr^jJ ii:crea8iii;< J.^ 
, cint that iVb averaging or>tr uii 
• •c-t*vr. n)i..»nv)6e, Lne hc?(inch'?.s feel 



251. I'm afraid I didn't realize how 

uncomfortable you currently are, 
Let me renew your medications 
and if you continue to have ^..^^^ems, 
let me know. 



251. Tl'.^.rs line. r€;rh;^pi! I can Ulk 
.::xur ccrUroi. (Thiv3 \Q Lio ( 



252. I apologize for angering you. I 
really do feel it would be safer if 
I went over your sjnmptoms first in 
case anything new may have occurred. 
What is the usual pattern for your 
headaches and how has it differed 
recently? 



252. NormAUy, i r.iight hnvo •^.♦>^o:'che 

"vorv *>^.'^nth« o: to, Cut for the l?/?t 
M.u.ai.c .1'^ li-cii inci t;«tSnvz to the 
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SELECTONLY WE. 

253. You*ve used up the codeine first. 253.. I'm of invinij to ^ over 

Would you like to talk about that? overyUv.r.c; plus your doubU too. 

not and !:avo no desire to be 
an aidict. I've been hurting: 
lately rj^.d updot witli everyon^j. 
(TLTIN C.TMEDIATnLY TO P.\GK 



254. It certainly does sound like your 
migraines are acting up. Why do 
you feel you need more Valium? 



254. I'v'j \)tikit\ rniHor'^blo from ihc^u 

'.'Vcrvoae. (TL'RN i:.::.:zDiATc:Ly 
TO ?AGF, :iO)' 



255. Frankly, Tm concerned about your 
increased une of Vallum and codeine. 
Can we talk about that? 



255. ;'ri tvrca of Jiav^ : p -v?r 
•^vorv t:an;r r'^^ your J.aili" ' 
r. : '^ji havf: WO rJeslro to 

ri:T:N lyr.AEMATEw: to page u> 



256. Frankly, your headaches concern 
me and I would like to give you a 
complete examination now to see if 
I can discover anything. If nothing 
turns up, we can renew your 
medications and have you call back 
if you're not better in a few weeks. 



256. MMM. { Mvrk thev r^rvi Uvj oc^n^t? h3 
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SELECT ONLY ONE. 



257. What's happening In your life that'a 
streasing you lately? 



257. .^rJiir^; ii\ i;articulca-. As a niaUcr 
ci faotj we have just rncvcd Lnlc 
iicw hcu3<j. Tiie cominiinicy in niu*.": 
c<=»tter than the one nrid been livinj^ 
.11 Ltiiforc movin;^- ijcrv. My hitabnnd'H 

pnc r:o6'uica. . . wiiy \vc C9ir,e 

i.cre in Uiii lirrt plrtco. ^T'*'^^^ 

:,ir.;cCiATii:LV to I'hge 



258. With everyone? 258. uml way, cue 1 

j\:i,^iU' \v\\h my })os'bar,G ar.d kid». 
•Tcbably it wiU :i\vav ruter we 
nre comp:(:t*iiy MiiiU c \ro\r. Ui^ 
;r:cve. .M:er ?,U, ;:\'3 oiuv uvea 



259. I'm sorry I upset you too. What I 
do need to determine Is how often 
you need to taicc your medicines. 



259. -.ioiy, it rt*:Tr\B like I am *nkinp: 

* time?' n dpy 'cr Ir^i r,^^^J)fl)J), 

♦ .'.♦.•ni, "?HC*i roc('.'iUy I'^^-n for tu'o 

r.r: tni-nnr: rver 10 Cr.:orrx:t ciJj u 
Wvck- T:;t: liust lew 'X'iiv\^ t'vt' ^^ne 
over 3 iutio. I've hcon cjt <:! tijv 
.■ri.*?ino w.r pbovjt bvo \^^^/t^t^ but I 
uun'i ioikc u cxcepL re rely Lt.'cauBe 1 
i^<.^ jici; tr^ my stcmrK'f. .,TLTI;\' 



260. I apologize for angering you. I 
really do feel it would be safer 
If I went over your symptoms In 
case anything new may have occurred. 
Perhaps for the time being I could 
renew your medications and we can 
talk next week when things have 
calmed down a bit. 



260. '/^wt r wr.a. ^ .jrh:^r«c» » cRJi ta-U: b»:t:i:r 



142 

ERIC . 



-104- 
SELECT ONLY ONE. 



261. How did you feel about It when 
he Informed you? 



261. Ha came in :md oroudly :u)nounce^ \:.\z 
promotion and salary raiae of 
^'^^^'^(^ y>lu."^ tiie move. It was pretty 
:iram,itic at the time. I i.uesa I was 
prnfttv excit«?d for him. (TUI7N 
IMMCOUThJLY TO PACb: 21)* 



282. Moving Is certainly most stressful 
on the woman ol the house. Valium 
may not be quite what you need to 
cope with what's going on. I'll 
prescribe something that's a little 
stronger. Check back with me in 
a couple ol weeks when things have 
calmed down. 



262. Thr\t'3 line. Probably it Aili all blow 
,a ti'iO ond >of cJhc case. LTRN 

.:.::.ni:DL\Tfc:LV TO vacje ;U) » 



263. Informed you that you were moving? 263. He came in and 'M-n'yilv 'i:\nojr.c:a 

iwwx Demotion v>n(i .salary raiije 

\ V5.'a:i, orotr.^ 'iXcUoci tor !um. 
\VjI\k^'^ *M\i vI:.'JiJiA TIlLV to rATjI-i 21)* 



264. Moving certainly is a stressful 
period in one's life. I'll renew 
your medications and if you continue 
to have problems, let me know. 



264. r\-.r3 fmo. P^^oabiy \t -ail ?'.ow 
o ' r.'J ;;a.*o TU^'N 
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SELECT ONLY ONE. 



265. rd like to give you a physical 
examination to rule out serious 
causes for your headaches. 11 
nothing turns up, then I think we 
might want to start thinking about 
having you talk with Dr. Gray, our 
psychiatrist, tc see if he might be 
able to help. Does that sound like 
a reasonable plan? 



265. \Vf if that's what you think is 
b<*3t. Set it up as quickly as 
pcd.sibie yo I can fjec my medicine 
r'^no'.vod (Tl*i3 i:; uhc end nf 



266, I could feel comfortable renewing 
your Cafergot now, but I do^.^t 
think I could give you Valiu'm/ai^d 
codeine. 



266. ,\ J«»ern t,") bo conuv.anu^-^uuic; 

-'Thi3 '.3 tho VDd of tlio cnJo. I'^^T^N' 
iUi^\iuVL\ r>:LY TO ?AGS 3iJ*' 



267, Upset with everyone? 



267. ' • ^Tis '%'>v. -HJt I ',\iLod 
Alter all,, U'3 onlv h<Mj:; n.'^ouc 



268, rm sorry you're angry. Are you 
often this upset? 



268. :: o ji::::J -i^ \: ; : , : \' i 

/ — -jly \: V ill ^ .r-.:\y rv.'.cr 'ir-j 
.^niptoiC'iy 3v:tiv:vt from -.ho r.:o*:'« 
^f''''r Ui, :t':3 oni'v b-'-'^n '^b'>uc 

: n o J n .J ' " p. t v \v • > r 'j r:: : > v - . 

""•r:^:^ r: to 
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SELECT ONLY ONE. 

289. 



u nt'f-^s ^h:U way, b^Jt I g\icsfl if'fl 
i;;Ci»iiy wicii n)y husband and kids. 

: ro v*ompif*u-Iv Pr-tlk:d from tiic 

.'•^.ti'S ••:i:.<:c u\y husband, 

.! f-T' that we were 



TO 



I I 



270. Vallum may not be quite what you 
need. Td like to give you aomethlng 
to help Hit your mood too. Why don't 
I talk it over with the doctor, got you 
the prescriptlonj, and see you again 
when we get these headaches under 
control. 



270. Th>>iikj. \ -ppreclato your hv-p. 

TUnN IMMnDL'MZLV TC PAGE aI)" 



271. Tm concerned about your Increased 
use of Vallum and codeine. Can 
we talk about that? 



271. I'm nrecl vi haw.*.': ovt-r 

cVf^ryUuaf; y^»jr ccubto tec. 
Vm not pnd have no desire tc be .-^n 
.'..uict. I've li:en hurting btoly 
",.d Ml iiet -.vili. ♦•v*-rvone. (Tuf^N 
'MNIKCLTHJLV TC P/.GS Ib)*^ 



272. rd like to give you a physical 
exammation to rule out serious 
causes for your headaches. If 
nothing turns up, we might want 
to start thinking about having you 
talk with Dr. Gray, our psychiatrist, 
to see if he might be able to help. 
Does that sound like a reasonable 
plan? 



272. '''' .i V'-U '-'^ ^.:Ht. 
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273. . , , and then all the work of moving. 
I think I umierstand what you must 
be going through| so let me renew 
your medications. 



-107- 

SELECT ONLY CWE. 

273. TI:at'3 fine. Probnbiv it ail 
Vlow over -uter I'm locuily jet^-L, 



^Ui^N fM:>;EDL\TELY TO PAGE 31)* 



274. Well, It certainly does sound 
exciting. Has It continued to 
be that good for both of you ? 



274. Oh, yo8. iio'3 v^r/v h:^oov :n hiJ 
^vorK nr.ci in our r.ew no^idy .vruelj 
is miicii bi^t^'iv ;.uin ch\; oU cno. 



275. For him? 



' ..'-'vor mu move cioprev3at-J .nd acis'ry,^ 
"r.:i r aon't V-nnv u;!»v. A.j I nr^io, o^: 

TO ?.\0Z 



278. What about for you? 



278. Vwu kur;Vv\ it'T ro-\llv ^♦M:*d, but 
.V.' : ^o.i c .vr,v. • jai,:. 
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SELECT ONLY ONE, 



277. That anger certainly could be 

caua Ing the headaches . Why don' t 
I see U we can come up with 
•omethlag better than Valium 
to help you get control of things? 
First, I'd like to do a physical 
examination to make sure 
everything is okay. 



277. Th.it sounds iine. I certainly ueea 
>xn^f--t!un<x. (This is the '»»>H of the 
c:iBe. (TURN IM.MEDIATIJLV TC 
PAGE 



278. Some people have more trouble 
accepting happiness than pain in 
their lives. Perhaps this is 
occurring? with you. If I find 
nothing wrong on physical 
examination, then why don't 
we think itbout having you work 
on this with Dr. Gray, our 
psychiatrist? 



278. V.'Uii, ;:' tl\?X's v.hai you ihlrM is best. 
Sot U up c.uirkly pcs?S'*^lo no 1 
\i\y vAc6\cAm rt.iicy.v-j. (ThiP 
ihe end cf tlie case- Tl'iui 
w.KvfEDLATELYTO PAGE 31)^ 



279. Vallum may not be quite what you 

need. I'd like to give you something 
to help lift your mood too- Why 
don't I talk It over with the doctor, 
get you the proscriptions, and see 
ycm again when we get these headaches 
under control? 



279. ThnVs fine, ''crhar;.? I '^?r' tnik 



280. You feel angry and depressed over 
the move and don't know why? 



280. "^'^^^ • sn.iuid bt' ^:jncv ^'nisc out 
. I irv no me town wliich \s ^^jUv^ 
1 jOC ;n\i(.s iro»*i j>^*c. T*"^ . i^vavs 
cA r.-jov'.p.'j n\v?.v trom ir.cro 
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SELECT ONLY ONE. 

281. Some people have more trouble 281. I <:^(iS^ VW oavt? to a^cu. 

accepting happiness than pain in '''t's see wnat yt.u />».d :r vr: 

their lives. Perhaps this ia -hv.^icai first. (This Is the enc cI 

occurring with you. K I find T?,n.:EDL\ Tli LY 

nothing wrong on your physical '^"O TACE :nl* 

examination, then why don't we 
think about having you work on 
this with Dr. Gray, our psychiatrist? 



282. And In splto of all this good fortune, 
you're having headaches. I guess 
that makes me even more Interested 
in having you see Dr. Brown, our 
neurologist, to make sure that some- 
thing more serious ifl not going on. 



282. V/cII, V^zV'i -hat you ihiiik le b^-Bt 
J<it it up as quickly t.z pn.^.sibl'j ; 
':rir5 ^'^^ n^v n^naicine reiit;\'.«:vl. '/P^'p 
IS Jie end of tiie case. TUHN 



283. What about ou? 



283. 



.1 . , ...... , . ' ^. / .J 

■*nv-' -1 u^w, 'Nui^er iioaii:. /ruixlx 



284. Well, \i sounds like it was a busy 
time, but now things should be 
settling down. So why don't I renew 
your medications after I check you 
and have you phone if there's any 
further problem ? 
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285. Mayb« you juat mlsa familiar 
placet and activities. 



SELECT ONLY ONE. 

28S* 



w^U. . . I \v?kJ completinf? my 
clUaartation in rnlcrobiolo^ 
xvh«n 'Jus movt* camtj up nnci 
now { foumi out I cnn't comploXe 
\[ d')wn hi* TO btfCRuae tlie univernUy 



288. Wasn't right? 



288. W'ju. . . I w^d con)plvtlag my 
•ih^jertntion in microblolo.37 
^h::n thi3 mavu cam<j up and 
'.o*A* I fcAHJd out r can't comploto 
u down here becnu30 the xinivtiratt)* 
o,;^.,*^r^'t offer that de^rn^ti. (TUPN 
IMNrKDLATKLY TO PAGK 25)* 



287. Mayb« you tetl that way because 

you weren't involved in the decision 
to move . 



287. ,VoU. . » I v/nn cornpUy'>p: my 
oijjcrtation in microbiolocv 
when tiuH mow*: r;imu up and 
'*nw I found out I can't complote 
t 'iown here because the univeraltv 
\ vi3iV( offer that degree. (TUHN 
;:;mfdia'it:ly to page 25)* 



288. What seemed to be wrong? 



288. -'-'U . . I ^'^s completiu^^ my 
M^doriat^-^n in irjic robioio>rv 
'.v^»»n Jui.i ^lovc up 'mJ 

.. r.; I fjur.J '^"t ! cnj\*t comoiete 
' j^—r: !; M'.r ^»'(':r::>o the \)fuv«*r:5'';' 
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SELECT ONLY ONE. 



289. How has that left you (eeUng? 



289. f .^e.Hd alaappointed^ hurt and anjji-y. 
(TUr:N IMMSDLATELY TO ?AGE 2S)» 



290. Aren't your children still pretty 
young and demanding moat of 
your time? 



:n 3chool. Mnvbo u»id w\ll hvlo 



291 . Couldn't you try something else ? 



291. I ' "i?"\:nt vnluntf;orjn,.( nt the 

^-^chool. r.tavDo tiiiS will help take 



292. You must be really angry at your 
husband (or making you move*. 
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293. What have you done to get over 
these feelings ? 



-112- 

SELECT ONLY ONE. 

293. 



I've Uiought ;ibout volunteering at 
U)e local hospital while my kids are 
in school. AlayLe tins will help to^e 
t:v rrAud off 'hings and jjive me some 
r.hvx to uj. ' rUHN IMMEDLMTLV 
TC TACC r"}* 



294. That*s probably accounting for your 
headaches. Have you thought about 
ways to cope with your feelings? 



school. ^'.lavbe ^IVis v/iil i>elp iixko 
•ur- to do. {T\T^J< }MMEDL\T2 V/ 



295, Angry at your husband? 295. ^'^^ ^^^^iv t^v 'ii-b-^nc, l.ui niv.seu. 



:^l^^KL.A'^Ii::.:l 



296. Angry at whom ? 296. ^ ' ' 'v • <^ pno, l:ui nnsoii. 
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SELECT ONLY ONE- 

297. That's a good Idea. I'll finish - 297. Thanks. I appreciate your help. 

your examination now and renew (This is the end of the case. TTON 

your medications. '.y,r\EDiATlZLY TO PAGE 31)* 



298. I think you should discuss your 
feelings with your husband. Til 
renew your prescriptions, and 
we'll make an appointment for 
next week to talk further. 



298. T!;;ui};s. I fippreciaU your i"?.^ 

JS end of the r;^5e i*LT5N 
'?.^M7.nTATp:r.Y TO r/'CK aI)^ 



299. Sounds like you're getting things 299. v---^- : • rcc:r,^e your :u-.... 

under control. Til finish up with ^^-^ -^^^ ' ^-'-^ case- Vl':\:^ 

the physical and then we'll get your > *Mr;j^i/> ro VA'jrl - 
medications renewed. 



300. Perhaps it would be a good idea for 
you and your husband to come m and 
we could discuss this further. In 
the meantime I'll renew your 
prescriptions. 
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SELECT ONLY ONE. 

301. Angry at yourseU? That must be 301. /^.-d. i^^. ...uo<: i artouicn i d-^ 

an uncomfortable feeling. , »o ^>^iri,^h, ^ut I ft^el like Tni 

.-'V.itK'd 'live rriy career too. 

uT.Dorcan: u .vas unUi now. 

vuHi; ^^:^^i:oi.^T£LY to ':'>1 



302. I guess I could see why you're 

angry at yourself. You could try 
something else. 



302. »'v- L!.:>'jt.jC :iouut voi\::r.,'k:T\ii^ r.^* 
''*noji. :«tiybo chlj wiU iiol) iJy 



303. All that anger certainly can give 303. i'v<i :nr)\iq;:t - :zvA \ jnc»"--',r^ nt 

you headaches. What do you think ivcr»i .*:oap, cal ^vniij '-jv/ '/.;di ur.*' 

would make you feel less angry? jr^ocl. M/ivbt- tJr,j will r.olp 'nk*^ 



304- That doesn't sound so healthy for 
you. Have you thought of any 
solution? 



304. . ji.^u.c.'i iV-'Ov:: ■.'oiun^ . ^ 
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SELECT ONLY ONE. 



.?05. Perhaps the children are 

jrown you can return to your career. 



305. rv(; thcuirnt about volunteenn^: at 

;r.»; locni noanitnl while mv VAdo are 
i.i soiKvol. Maybe tl\is -^"xli help take 
n.V ..iir.w Cif fiun^rs and ^ive r.\e 
^o;--u.:/r to Co, (TURN ^M^:KDIJ^TELY 



306. Perhaps you can try something else- 



306. ' hy::r.c.j'y- 

. < : . " 'VN.' 'Ms \v»ll : rl.) * 



307. I can certainly see the bind- I'll 
finish the physical and then renew 
your medicatioag to help you cope. 



307. 



t . • r cKu'* v^.m; Lr^v* 



308. Do you and your husband talk over 
majcr decisions like a move? 



308. 
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SELECT ONLY ONE. 

309. We will have to he endlag for now. 309. Thr^nks- i :^pprec\ate your h^ip. 

In the meantime I wonder if you, (TAs is iJiu* ond of the cnse. TURN 

might not try being more assertive VM^\^:dL\ r£LY TO PAGr: Cij* 

at home' 



310, Perhaps if you and your husband 310. ^\,<. { -^-ryrccir^n yu^i 

come in, wo could work on this !^ • ^ ^^-^^ oi Lho c - 

togrether. IMMEDU ri=: TO ^AGK 



311. I'd like to talk with you more next 311. . ; v^-^ ;..^(-, 

week. In the meantime you might think ; ij u>rj oi "jv? '^ps'/ T- 

about numerous alternatives, including r-iK-...- — / r -t - - ^) PM;,.r > 
separation. 



312, Wiy don't you talk this over with 312. ; 

your husband and let me know how , ..o \s . 

things work out by phone next week? -iMTM'N' 
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HrySICIAK*3 ASSISTANT TPS OBSyKVATIONAL rOtOI 





1. 


]Hm~caa« p«rtln*nt, open end«d. 




2. 


Ca«« pertinent, open ended. 




3. 


Koo-CAM p«rtln«nt» closed (»et of possible knmw^rm Is predictable). 




4. 


CsM pertinent, closed (set of possible unswers Is pr^dlctsble) . 




5. 


C8s# pertinent, Inspproprlste lesdlnf question. 




A 
P* 


lietxecvs psvicnw cowenb or ^\Jv■bAua nv • yr%jvw% 




7. 


Repetitious question. 




8. 


Continuous questlonlOf* 


Resctlocs to 


9. 


Inappropriately Interrupts pstlent/sbrupt shift In focus. 


Patient 


10. 


Encourages patient to continue talking. 


B«hairlora 


n. 


Uses technlcsl terwlnolofy. 




X A * 


IVtl 1 nwa.im nArti&An^ f*\ymm (n^w Inf omfttion^ * 




13. 


Falla to follow-up pertinent cuea (new Information). 




M. 


Clarifies own prior questlon/coment (e.c*» **«hat I really mean la**). 




15. 


Repeats own question verbatim. 




16. 


Appropriately suinir.rl£«8/8yn the sizes . 




17. 


Inapproprl ately suoimar Izes/syntheslres. 




18. 


Expresses understanding. 




19. 


Responds with Inspproprlste effect (e.f.^ disinterest. Judgment, etc.). 


Provl sloQ 


20. 


Reassures patient appropriately. 


of Svipport/ 


21. 


Reassures patient Inappropriately. 


Infomstlon 








2^. 


Olves ad vice/ Instruct Ions spproprlately. 




23. 


Glvea advlce/lnstinictlons Inspproprlately . 




24. 


Answers pstleat*s questions directly. 




25. 


Answers p«tlent*s questions evsslvely. 




26. 


Provides approprlste Information. 




27. 


Provides Inappropriate Information. 




28. 


Expresses empathy. 



8E9UEMCE OF ACTIVITIES 



I 


II 


III 


IV 


V 


VI 


VII 


VIII 


IX 


X 


XI 


XII 


XIII 


XIV 


















































































































1 
i 

i 
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APPENDIX C 



AUDIOVISUAL SIMULATION STUDY 
Patient Protocol Form 

The following categories are presented in order to assist you in developing 
a patient protocol for the pilot studies in interpersonal skills. The information 
you provide will need to be sufficiently detailed so that an actor or actress 
can develop the appropriate characterization * Particular attention should be 
given to data which are of special relevance to the evaluation of this patient 
by the P. A. 

These categories are meant to provide guidelines only. You may add other 
categories or omit any outlined here as you think appropriate* In developing 
a protocol for each patient, please organize the data under headings of some 
kind. This will assist us in identifying any additional work that may be 
needed on the protocol, and will als^ aid the actor in his assimulation of 
the role. 

I. PERSONAL INFORMATION 

Name 

Sex 
Ago 

Bl rthdate 
Birthplace 

Marital Status 

Number of Children (their ages and sex) 

Re 11 g ion 

Occupation 

Spouse's name, age, occupation 
II. CLINICAL SETTING 

Initial patient visit or follow-up? 

Purpose of interaction (primarily history taking?, counseling?, explaining?,) 
Patient problems (those that precipitated the present need for health care) 

157 
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III. PHYSICAL & EMOTIONAL HISTORY 

Medical history 
Drug history 
Psycho-social history 

IV. CURRENT PHYSICAL & EMOTIONAL STATUS 

V. PAST & CURRENT LIFE SITUATIONS 

Childhood family life & experience 

Adult family life & experience 

Current life situation and relationships 

VI. AFFECTIVE PROFILE 

VII. Appropriate P. A. behavior in response to patient's affective profile 
acceptance, understanding, reassurance 

VIII. Type of information that should be provided to patient 

IX. Type of Information that should be elicited from the patient 
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>kTiONAL Board or Me'dicai. Examinj^rs 



TONSILLOPIlVnYKGITIS, TONSILLITIS OR PHARYNGITIS 



Records to be \b3tracted : The following criteria are appropriate for the 
Initial work-up of a patient with this condition. 



HISTORY 



1. Symptoms : 



Acceptable: Sore throat, swollen throat 

Painful or difficult swallowing 
Headache 

Coated or **strawberry*' tono:ue 
Anorexia 

No associated symptoms 

2. Duration of Symptoms : 

Acceptable: Sore throat x days 

Malaise x 24 hr 

3, Mononucleosis (if under 30 years of age) : 

4» Current Medications : 

Acceptable: ASA 

Antibiotic 
No Rx 

6. Previous Drug Reactions if Antibiotics are Prescribed ; No manifestations 
are necessary. 

Acceptable: None 
NKA 

Allergic to : 



Penicillin allergy 

PHYSICAL EXAMINATION 

6, Throat Findlnrts : Evidence that throat was examined. 

Acceptable: Thro^.t inflamed, injected 
Tonsils inflamed , injected 
Exudative findings 

Tonsillitis, etc* under physical findings 
Throat normal 
Throat red 

P,A* Val 

3/2/76 15JJ 
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7. Temperature Recorded : 

Acceptable: (Listed in degrees) 

Unacceptable: Patient lebrilo 

Ears ; For patients under 25 years of age, HOT APPLICVDLE if patient 
is 25 yrs, old or older* 

Acceptable: Ears normal 

TM's normal 

TM*s not inflamed, injected 
Rest of EKT normal 

9, Lymph Nodes : For patients under 25 years of ago. NOT APPLICABLE if 
patient is 25 yrs. old or older, A global description or any reference 
to head and neck (cervical and/or submandibular nodes) • Mention of che 
nodes outside the head and neck Is insufficient by itself. 

Acceptable: Anterior or posterior cervical tender, swollen, prominent 

Adenopathy 

JilANAGEMENT PLAN 

10, Correctness of Action Concerning: Druo: ; The correctness of the action 
taken is determined by the pattern of response to item 6 above and the 
ordering of a throat culture. Appropriate patterns are presented in the 
table below. Providing the patient with a temporary prescription, for 

2 days or less, pending the return of the throat culture is an acceptable 
behavior, A refillable prescription or a notation that the patient will 
be told to stop the medication if the TC results are negative is also 
acceptable* 

Patterns of responses not listed in the following table or described in the 
above paragraph are considered inappropriate and the "NO" column for item 10 
•hould be checked. 



Throat 


TC 


TC 


Indicated 




Findings 


ordered 


results 


Therapy 


Duration 


+ 


Yes 


+ 


Pc 


10 days 


+ 


Yes 




None 


NA 


+ 


No 




Pc 


10 days 




No 




None 


NA 




Yes 


+ 


Pc 


10 days 



It the duration docs not appear in the record, and if no statement regard- 
ing future patient contact upon return of the culture is present, assume that 
the drug has been given for 10 days and check the "NO" column for item 10. 



P. A. Val, 

3/2/76 
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2j-!i£- Acceptable dnir:s include penicillin - G or V, erythromycin, 
bicillln. Other dru^s are acceptable if Justified by allerpios, 
^ additional organisms cultured, or Pc failure. KOT APPLICADLE if no 
drug prescribed. 

12. Dosage : NOT APPLICADIJ: if no drug prescribed. 

13. Duration : NOT APPT/ICABLE if no drug prescribed. Drug must bo pre- 
scribed for 10 days if no culture was taken or if the results are 
positive. If the culture is not back, the prescription should be 

for 2 days or less or some notation regarding contact with the patient 
upon return of the culture should be made. 

The following examples illustrate the appropriate coding of items 
10-13, 



Example 1, 




Y 


N 


UC 


N/A 


KID 


Throat red. TC to lab 


10» 


0 


□ 


□ 


□ 


p— < 

□ 


Rx Benylln Dinetapp 


11, 


□ 


I 1 

□ 


□ 


0 


□ 




12, 


□ 


— - 

□ 


□ 


0 


□ 




13. 


n 


n 


n 


LU 


n 


Example 2. 






N 


UC 


N/A 


WD 


Throat is hyperemic. TC taken 


10. 


□ 


0 


0 


□ 


□ 


Given Pen, G 400 for 7 days. 


11. 


0 


□ 


o 


□ 


□ 




12. 


0 


o 


o 


□ 


□ 




13. 


□ 


0 


□ 


□ 


o 


Example 3, 




y 


N 


UC 


N/A 




1/2/75 Tonsil hypertrophic L 


10, 


□ 


0 


o 


0 


a 


hyperemic , Rx: V-cillln 


11, 


0 


O 


□ 


□ 


□ 


500 k TC done. 


12, 


0 


0 


□ 


□ 


□ 


1/6/75 rc neg. 


13, 


□ 


0 


□ 


□ 


□ 
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ANGIKA 

Records to be Abstracted : The following criteria are applicable to the initial 
work-up ol patients with angina. 

Chart Selection or Dtn<>no3is; The following list of siCT out diacnoses Is! meant to be 
•uggeative only and not inclusive. Sign out diagnoses may vary considerably and 
still be acceptable for this study. A chart must show either chest pain or angina 
to be acceptable for the study. If the diagnoses listed as numbers 3-7 below are 
present on a chart, they are acceptable as long as they are (or have been) ac- 
companied by a notation regarding chest pain and/or angina. 

1. . History of exertional chest pain 

2. Angina Pectoris 

3. Coronary Artery Disease 

4. Ischemic Heart Disease 

5. Coronary Arteriosclerosis 
€. Aortic Stenosis with Angina 
?• Coronary Insufficiency 

HISTORY . . , 

1 • Character (quality) of Pain : 

Acceptable: Any qualitative description such as pressing, squeezing, 

etc. 
Dull, heavy 
Constricting 
Pressure 
Tearing 

Unacceptable: "Anginal pain" 

2. Location or Radiation : 

Acceptable: Substernal (any anatomic description) 

Anterior chest 

Retrostei^nal (c or s radiation) 
Radiates to neck, jaw 
left 

both amis 
Into back 



P.A. Val. 
3/2/76 

162 



ERIC 



I I 



3« Duration of Pain ; 

Acceptable: 30 seconds to a minute 

1-2 minutes 
About 3 minutes 

4» Precipitating: Factors ^ Mode of Onset ; 

Acceptable: Brought on by 1 flight of stairs 

On effort, during tenso periods 
Brought on by emotion only 
After meals 
While lying dovm 
With intercourse 
During day at work 
Whenever I exert myself 
Gradual, with exertion 
Sudden, with increasin^c intensity 

5, Asaociated Cardiac Symptom? : Pret*enco or abi>ence of symptoms. 

Acceptable: Accompanied by shortue«8 of breath (S,0,BO 

Denies associated symptoms 
PND ■ - - 

Diaphoresis 

Weakness 

Palpitations 

Anxiety ' " " 

Orthopnea 

Hausea and vomiting 



6, Time of Onset or Duration of I llness: 

Acceptable: When patient was 30 years old 

9/15/71 
Never before 

Increasing frequency in past 6 months 

10 years 

6 months ago 

First time yesterday evening 

7. Frcquoncy of Ant^lna : 

Acceptable: Increasing frequency in past 6 months 

2-3 times per week 
2-3 times per day and at night 
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B» Personal History of Smokintr; Specific Quantitation 

Acceptable: i ppd 

Non-sznokor 

Unacceptable: Smoker (insufficient by itself) 

9. Personal History' of Elevated B>P> : 

Acceptable: Denies elevated B.P. 

10* Personal History of Diabetes ; 

Acceptable: Denies diabetes 

Diabetic x 4 yr. 

11* Personal History'of Elevated Lipids ; 

Acceptable: Lipids or cholesterol 

12* Family History of Heart Disease : 

Acceptable: Family hx negative 

13, Activity Level of Patient ; 

Acceptable: Sedentary office worker 

Construction worker 
Jogs daily 

♦ Current Medications : 

Acceptable: None 

List of medications 

15, History of Obesity if the Patient is Obese : See general description of 
patient in P.i),* to determine if patient is obese, if some mention is net 
Biade in Hx. NOT APPLICABLE if patient is not obese. 

Acceptable: Overweight all ox Aife 

Gained 50 pounds in last 6 months 
(Recent weight change) 

16. Emotional History: Any reference to emotional status or life stresses such as 
work, death, divorce found in the review of systems (ROS). social history (SH) , 
or history of present illness (HPI) is acceptable. 

Acceptable: Anxious 

Handles stress well 

Has occasional depression 

Recent death in family 
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PIIYSICAL EXAMINATION 



17. Sex and \yelght ; Weight in pounds. 
Acceptable: 160 Xb. male 

18. Nutritional Status: 



Acceptable: 



19. Blood Pressure: 



Acceptable: 



20. Pulse Rate: 



.Acceptable: 



21. Fundi: 



Acceptable: 



22. Cardiac Size : 
Acceptable: 

Unacceptable: 

23. Cardiac Rhythjn : 
Acceptable: 

Unacceptable: 



Obese 

Well built 

Thin 

WN 



140/86 

220/136 

220/120/60 



76/inln 

130/mln 

76 



Normal 

Grade 1 A.S^ (arteriosclerosis) 
Arteriolar narrowing bilaterally 
Marked Increased arteriolar reflex 



Not enlarged 

Description of PMI (point of maximal lmpul8e)locatlon 
Heart normal 



Regular 

Normal Sinus Rhythm (NSR) 
Irregular 

Heart normal 
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24. Cardiac Auscultation ; Sounds - normal or describe quality. 

Acceptable: No (8> 

Gallops - description 

Description of (J) 

No rubs, gallops 

1st and 2nd heart sounds normal 

Normal heart sounds 

Unacceptable: Heart normal 

25. Lungs - only auscultation required : 

Acceptable: Clear to PkA 

Bilateral basilar rales 
Lungs normal 
Chest clear 

26. Carotid Pulses : 

Acceptable: +2 

Normal 

27 . Peripheral pulses - quality of pulses : 

Acceptable: Presence or absence of bruits, lags 

Pulses intact 
(Diagram o'*' pulses) 
Bounding pulses pre*5f;nt 
Diminished pulses 

LAJBOP^^TORY AND DIAGNOSTIC STUDIES 

If one or more of the following studies is found 
under Plan , it is acceptable and the appropriate 
criterion should be checked '*\TiIS", 

28. ECG ; Description of reading - an EGG alone w/o description i 

Acceptable: Evidence of old M.I. 

ST segment depression 
ST segment elevation 

29. Hemor.lobln or Hematocrit : 

Acceptable: Normal 

Hematocrit 43Z, Hemoglobin 14 g/ml 
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30. Blood Glucose <su^ar): 



Acceptable: Normal 
SUA-12 

31. Lipid s; Cholesterol, Lipid Profile, Triglycerides - Any one of these is 
Acceptable, 

Acceptable: Lipids elevated for age 

Elevated triglycerides 

32. Chest X-ray : PA view and interpretation to R/0 rib fracture, embolus i 
pneumonia Infiltrate, tumor, pneumothorax, CHF; 

Acceptable: Normal heart size 

MAKAGE^IENT PLAN 

33. Dletai-y Instructions if Obese, Hypertensive, or Diabetic : NOT APPLICABLE 
if patient is not in one of these three categories. 

Acceptable: Patient instructed on diet 

Referred to dietician 
Specific low calorie diet 

34. Patient Education: Instruction and counseling regardir^g such things as 
sacking, risk factors, and/or exercise. 

35. Medications : A listing of medications is acceptable. 

Acceptable: PRN Nitroglycerine 

: TG gr 1/150 PRN 
Isordil 10 mg q 6 hr 
Inderal 20 mg qid 

36. Follow-Up Appointmont ! A note that an appointment has been made and the 
date (e.g., 2 weeks, 7 months) is required. 

Acceptable: RTC 1 month 

Ro 1 month 

Unacceptable: Return PRN 

(No reference to return visit) 
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VALIDATION STUDY OF THE }975 PHYSICIAN *S ASSISTANTS 
CERTIFYING EXAMINATION 

INSTRUCTION MANUAL FOR RECORD REVIFV^RS 



Introduction 



The purpose of national examinations for licensure and certification is to identify 
those candidates who have not yet achieved minimum acceptable levels of knowledge 
and skill required to deliver adequate health care. If examinations of this kind 
arc to achieve their purpose, the content and format should be relevant to those 
functions the health professional will perform in practice* Moreover, there should 
be some positive relationship or correlation between a candidate's performance on 
the certifying or licensing examination amd his actual performance in practice. 

To date, however, national examinations in medicine and the allied medical profes- 
sions have not been validated by comparing examination performance with subsequent 
perlormauce in practice* Vdlidation studies of this kind have not been conducted 
primarily because feasible evaluation methodologies for accurately and reliably 
assessing day-to-day performance have been lacking* Recent experience with medical 
record audits suggests that they may be a useful tool for this purpose ♦ The use of 
the medical audit is based on the assumption that the entries made in the medical 
record reflect the care provided to the patient and/or the patient's family by physi- 
cians and other health t^are personnel. This assumption is consistent with the long 
standing purpose of the patient's medical record, namely to document the care provided* 

While it appears that a review of medical records can be used as a methodology for 
evaluating some aspects of day-to-day performance, it should be recognized that thex^e 
is a body of knowledge, skills, and attitudes relevant to the competence of health 
professionals that the medical audit cannot assess ♦ For example, it is difficxtlt to 
determine from the medical record the degree of the physician's empathy for the 
patient, the accuracy of the physician's skill in palpating an abdomen, or the pa- 
tient's willingness to comply with various treatment regimens* These aspects of 
physician functioning are difficult or impossible to assess with the usual paper and 
pencil test as well. For this reason, it should be anticipated that the physician 
assistant's performance as measured by the medical record review will correlate with 
only some portions of the Certifying Examination and that neither methodology will 
measure all relevant aspects of performance. 

P^ir pose and Method 

The overall purpose of this validation study is to determine the extent to which 
performance on the Physician* s Assistant Certifying Examination is predictive of an 
individual's subsequent performance in actual piactice as measured by a medical 
record review* 
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In developing the protocol for this study, the NBME project staff asked a group 
of physicians and physician's assistants to select a number of clinical problems 
which physician's assistants frequently encounter and then to specify the informa 
tion necessary for the diagnosis and management of patients with each of the 
clinical problems. 



Responsibilities of the Reviewer 



Re t ri eving and Coding Medical Records: 



The reviewer is responsible for obtaining the records to be abstracted from a 
member of the clinic staff and completing a coding form for each record 
identified by staff personnel as appropriate for this study. 

In abstracting data from a chart, you must determine whether data have been entered 
in a patient's medical record which satisfy the previously established criteria. 
For each criterion, you must check one of the following options on a coding form: 
YES if the medical record entry indicates that the criterion has been met; NO if 
the criterion has not been met; LTNCERTAIN if you are uncertain if the criterion 
has been met (because of problems of legibility or interpretation of criteria); 
and NOT APPLICABLE if the criterion is not applicable for the situation at hand. 

A fifth column, MD, has been added. you can distinguish between PA and MD 

entries and note that a specific piece of information has been entered by the 
physician, check the column labeled ''MD" in addition to the YES column. If you 
leave the "md" column blank, it means the entry was made by the PA. If you cannot 
tell who entered the information in the record, check the box appearing at the end 
of the coding form labeled I could not distinguish between MD and PA entries on 
this record ' . 



Confidential ity : 

Rules rcg'^rding the handling of Medical Records are of the utmost importance. 
The following apply to this study: 



1. Never open a patient*s Medical Record unless it is a patient on your 
list. When a record is reviewed, do so in a manner such that the 
record cannot be seen by patients or office employees. 

2. Never discuss the contents of a Medical Record. 

3. Never leave a Medical Record unattended where it may be seen by 
patients or others. 

4. Treat all information contained in a Medical Record in a strictly 
confidential manner.. 
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Do*g and Donets for Reviewers : 

!• Where the initial work-up for a diagnosis, such as hypertension or 

diabetes mellitus, was completed in the hospital, use the appropriate 
set of follow-up criteria in relation to follow-up evaluations of the 
patient in ambulatory care settings. 

2. Sets of criteria for intial work-ups should be applied only to patients 
seen in the office or clinic for their initial evaluation. 

3. Try not to use casual drop-in visits. An example of such a visit would 
be the visit of a patient who is Just passing through the community of 
the practice. It seems rather unlikely that such a patient will return 
to the practice and, therefore, the criteria developed for this study 
are inappropriate. 

4. If patients are new to a practice, the selection of the appropriate set 

of criteria depends on the availability of information collected previously 
by another practitianer . If there is a notation in the record for the 
initial visit that prior medical information is to be obtained, apply 
the appropriate set of follow-up criteria to the initial visit. If prior 
medical information has already been obtained, use the set of criteria 
for the initial visit and accept all data in the record that satisfy the 
criteria. If there is no indication that the practitianer will seek prior 
information, the set of criteria for the initial visit is to be used. 

5. Study the criteria carefully immediately before befinning to review any 
medical records in a given medical condition. 

6. Review all charts for a given medical condition before going on to another 
medical condition. For example, review the criteria and instructions for 
hypertension. Collect all patient records that are to be reviewed for 
hypertension and review all records before beginning to review the criteria 
for another medical condition such as health maintenance for adults. 

7. Use pencil to code, to facilitate corrections you may wish to make. 

8. Mark only one category (YES, NO, UNCERTAIN, or NOT APPLICABLE) for each 
criterion title. 

9. If you can determine (usually by handwriting) that the entry was made by 
the physician, mark the MD category in addition to the YES or UNCERTAIN 



10. If age is missing from a medical record, mark the UNCERTAIN column for all 
•ge dependent criteria. 

11. If information is present in the chart but the information is not applicable 
for a patient of that age, mark the NOT APPLICABLE column. In other words, 
NOT APPLICABLE takes precedence over actual chart entries. 



column. 
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Reliability of Abstracts ; 

In order to make equitable Judgments about performance on the basis of the medical 
record audit, there must be consistency in the manner in which the record entries 
are Judged and entered on the coding forms. Thus, if several reviewers code the 
same records, their coding form should be consistent with one another. If a re~ 
viewer reabstracts the same set of record entries on two separate occasions, the 
reviewer's entries on the second coding form must match the entries made on the 
first record review. In addition, reviewers working at one site should interpret 
the c rlteria in the same way as reviewers working in another region. 

This manual has been prepared to help you make reliable Judgments. On the next few 
pages you will find some rules and procedures that apply to all problem areas* 
Following the general rules is a section specific to each clinical problem. Each 
set of problem specific criteria includes a list of criteria (e.g., Duration 
of Present Illness, Symptom of Sore Throat, etc.) and criteria instructions . The 
criteria instructions provide you with guidelines about how the criteria 
should be interpreted, examples of acceptable and unacceptable entries, and nota- 
tions concerning the applicability of the criterion. The examples of acceptable and 
unacceptable entries are not intended to be exhaustive lists of chart entries. A 
P. A. may use any of a large number of synonyms to express his/her impressions. You 
must refer frequently to tne instructions when abstracting records. The criteria 
items filone are not sufficient to ensure the reliability of the judgments being 
made. ... 



Obtaining and Submitting Coding Forms: 
Mail all completed coding forms to: 



Mrs. Arleen Caba 

National Board of Medical Examiners 

3930 Chestnut Street 

Philadelphia, Pennsylvania 19104 



All requests for blank coding forms should also be directed to Mrs, A. Caba. 
Please make these requests in sufficient time to allow for duplicating and mailing. 

Please submit all completed forms immediately after completing thera. Include your 
Time Reporting Form and the office report form with your completed coding forms. 
Envelopes are available from Mrs. Caba. 



Problems and Questions: 



Any questions pertaining to the criteria and the medical records should be referred 
first to Dr.. Vivian Erviti at the NBME. If Dr. Erviti is not available, questions 
should be referred to Ms. Jane Bunce. If neither Dr. Erviti or Ms. Bunce is avail- 
able, please leave your message with Mrs. Arleen Caba. Dr. Erviti or Ms. Bunce 
will return your call as soon as possible. 



Vivian Erviti. Ph.D. 

National Board of Medical Examiners 

3930 Chestnut Street 
Philadelphia, Pa. 19104 
(215)349-6400, Ext. 271 
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12. Review the coding form to ensure that til criteria are completed and all 
identifying infonnation has been entered. In completing the top of the 
coding forms, be sure all data are right adjusted, i.e., are placed in the 
blank spaces farthest to the right first. If the site you are visiting has 
been assigned »^15, place this number on the lines below which "4" and ''s'* 
appear. If a log or patient I.D. is 1049, place these figures on the lines 
11, 12, 13, 14. Leave the lines numbered 6-10 blank. Dates should be 
entered as 9 2 7 5 for September 2, 1975. Enter your assigned number 

15"^ 18 20 

in spaces 34-36 » When a coding form has more than 1 page, be sure to com- 
plete the top of the second and succeeding pages. 

13. If an item is NCTT APPLICABLE because of age or sex, mark N/A even when an 
entry for the item exists. 

14. Explain why you have marked the UNCERTAIN category for any criterion title 
(a note beside the title, at the top or end of the coding form will be^ 
satisfactory). This will aid the NBME staff in identifying problems with 
the criteria. 

15. Give an estimate of the time it took you to abstract a jrecord at the end of 
the coding form. 

16. Prepare a separate listing for records pulled that cannot be used for 
abstracting. For each no i-abst ractable record pulled, enter the problem 
area and the reason for not abstracting the record. 

17. Place all additional written comments at end of coding form or on a 
separate sheet of paper. These comments will aid us in the revision of 
criteria and the subsequent preparation of manuals for record reviewers. 



Calculating the Age of Patients : 

Some sets of criteria include several titles that are age-specific. Therefore, 
at times it will be necessary to obtain the age of the patient in order to 
determine whether or not a title is applicable to the record being reviewed. 
Age to the nearest year is sufficient. 

If the age is missing from a chart being audited and there are age-specific 
criteria, audit the chart but mark each age-specific criterion UNCERTAIN. 
Note on the top of the coding form that the age was unavailable. In certain 
cases where age is missing, another entry in the record may make the general 
age range obvious. In these instances age-specific criteria can be checked 
either YES or NO. Please note on the top of the coding form the phrase used 
to determine the age range. An example in this latter category might be a 
notation that the patient is retired. Obviously, such a patient is above a 
35 or 40 year old minimum. 

Interpreting the age of Patients : 

The interpretation of NOT APPLICABLE under 25 years of age will be that the 
criterion should be satisfied for a patient exactly 25 years old and older, but 
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would not be applicable for patients even 1 day less than 25 years old- The in- 
terpretation of NOT APPLICABLE after 25 years of age will be that the criterion 
will not apply for patients exactly 25 years old and older. 



Interpreting the Criteria : 

Entries found under the Review of Systems (ROS) are acceptable for history 
Items. 

A statement indicating either the presence or absence of any of the factors 
listed as examples of appropriate chart entries is acceptable because statements 
in either direction are valuable. Thus "no fever", for example, should be 
checked YES. 

As stated previously, the lists of acceptable entries following each item in 
the criteria sets represent a few of the many ways in which signs and symptoms 
can be combined with other descriptive phrases. The lists are not all inclusive 
and the abstractor should check an item "YES" if any one of the key phrases 
appears, regardless of the presence or absence of additional detail that might 
accompany it. Please re^er frequently to the instructions when abstracting 
records . 
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INTERPRfTTATrON OF LABORATORY DATA* 



Simultaneous Multlspecimen 
Analysis (SMA) 



SMA-6 



SMA-12 



SMA-C-20 



Sodium 
Potassium 
Chloride 
BUN 

Qlucose 



Total Protein 

Albumin 

Globulin 

Inorganic Phosphatase 

Bilirubin 

SCOT 

LDH 

Alkaline Phosphatase 

Calcium 

Glucose 

Cholesterol 

BUN/Cretinine 

Uric acid 



(all entries for 
SMA-12 plus the 
following: ) 

SOFT 

CPK 

Iron 

Triglycerides 
Total Protein 



(NOTE: Urinalysis (U/A) — grams percent (gm %) or 4 
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p. A, VALIDATION STUDY 



ANGINA FOLLOW~UP VISITS 3^ 7 

1 2 



P,A.# 



3 4 5 



Log 0 or Pt. I.D, 



6 



14 



Pt. Sex _ Male (1) 

Female (2) 

27 

Date of Abstracting Mo Day Yr 



Date of Patient Mo Day Yr 
Visit 



29 31 33 



16 18 20 



Abstractor 



Age of Patient _ _ _ years 
21 22 23 



34 35 36 



No, of Abctrict ^ First (1) 

Recheck (2) 

37 



1. Complaint s: 



2. Blood Pressure : 

3. Pulse Rate : 

4. Cardiac Auacultation : 

5. Cardiac Rhythm : 

6. Chest Ausciil tation: 



HISTORY 



PHYSICAL EXAMINATION 



Y N UC * K/A MD 
□ □ □ □ 038 



MANAGEMENT PLAN 



7. Continued Management Plan: 



□ 


□ 


□ 


□ 


□ 


□ 


□ 


□ 


□ 


□ 


□ 


□ 


□ 


□ 


□ 


□ 


□ 


□ 


□ 


□ 


□ 


□ 


□ 


□ 


□ 


□ 


□ 


□ 


□ 


□ 



CD I could not distinguish between MD and PA ^n^rles on this record, 



73 



Time required _ mlns. £ ^ j 1 
74 ~ 76 77 78 79 80 
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